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PREFACE 



This Report is the third element of the Rand/HEW study of performance con- 
tracting in education. The first was a Report by J. P. Stucker and G. R. Hall. The 
Performance Contracting Concept in Education . The Rand Corporation, R-699/1- 
HEW, May 1971, which discusses the theory behind performance contracting and 
how it was applied during the 1970-71 school year. The second element comprises 
six volumes, which examine eight performance contracting programs in 15 schools. 

1. R-900/1-HEW, Conclusions and Implications . by P. Carpenter 
and G. R. Hall 

2. R-900/2-HEW, Norfolk . Virginia . by P. Carpenter 

3. R-900/3-HEW, Texarkana . Arkansas and Liberty-Eylau . Texas . 
by P. Carpenter, A. W. Chalfant, and G. R. Hall 

4. R-900/4-HEW, Gary\ Indiana . by G. R. Hall and M. L. Rapp 

5. R-900/5-HEW, Gilroy \ California . by M. L. Rapp 

6. R-900/6-HEW, Grand Rapids ; Michigan . by G. C. Sumner 

The present Report, distilled from Rand’s findings in both previously reported 
and new research, is a Guide addressed to school board members, administrators, 
and other decisionmakers involved in school district affairs who may be contemplat- 
ing a performance contracting program. It delineates questions, issues, and choices 
they are likely to confront. 

The Guide is published in two parts. This volume discusses the planning, opera- 
tion, and evaluation of performance contracting. The Technical Appendix. R-955/2- 
HEW, goes into more technical detail on problems of test and measurement and of 
cost analysis associated with performance contracting programs. It also reproduces 
the contracts involved in eight programs. 
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SUMMARY 



An educational performance contract is an agreement between a learning sys- 
tem contractor (LSC) and a local educational agency (LEA) for instruction of a 
specified group of students, with payment according to measured achievement. LSCs 
have included both teacher groups and private firms, but the latter have been far 
the more numerous and receive the focus of attention in this Guide. 

The Guide neither endorses nor opposes performance contracting. Its purpose 
is to assist LEAs in assessing the major planning, management, and e\aluation 
issues associated with this type of program. Many issues are common to all new or 
innovative programs, but three areas are especialiy important for performance 
contracting programs: legal issues, program management, and measurement and 
validation of student achievement. 

Despite the large number of programs to date, novel legal questions still attend 
performance contracts. Because stale and local education codes, procurement regu* 
lations, union contracts, and employment arrangements differ among LEAs. each 
LEA must conduct specific research to determine rights, responsibilities, and proce- 
dures. The Guide discusses some basic legal matters that a-e likely to interest all 
LEAs, however. 

Performance contracting programs require active LEA program management. 
In the initial phases the LEA must assess education needs, prepare the Request for 
Proposal, select the contractor, and write and negotiate the contract. When the 
program is underway, the LEA management must monitor the program and resoKe 
questions promptly, and effectively. Finally, the LEA must assure that the e\alua- 
tion and assessment of the program generates data that not only meet professional 
standards c f reliability and validity, but provide decisionmakers with the informa- 
tion they require. 

This last point introduces the third topic of special interest, the measurement 
and validation of student learning. This is the key element in performance contract- 
ing. With the contractor s payment hinging on student learning, measurement must 
be both reliable and valid. In turn, accuracy and objectivity on the part of the 
evaluator take on extreme importance. 

Validation is an audit of the appropriateness of the program’s evaluation proce- 
dures and verification of the results as reported by the project evaluation. Ideally, 
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it should include a review and critique of the criteria for test selection, an assess- 
ment of' the reliability and validity cf the tests as measures of the student learning 
desired, inspection of the conditions of test administration, checks of test-scoring 
procedures, and checks of procedures in computation for payment. A well-planned 
and well-executed validation of student achievement should be an integral part of 
any performance contracting program. This Guide devotes several sections to valida- 
tion and evaluation issues, and the Technical Appendix goes into further detail. 

The Guide follows the life-cycle of a project. Three major phases are ident.fied: 
planning and contracting, operation, and evaluation. Within the planning and con- 
tracting phase, six activities or events are discussed: (1) legal research, (2) assess- 
ment of needs, *3) Requests for Proposals, <4) proposals. <5> selection of LSC and, 
possibly, other program participants. (6) determination of contracts. 

For the program operation phase, the Guide considers: <1) selection of' schools. 

< 2 J personnel selection, <3> teacher training, (4> student selection and transfers, fa) 
program monitoring, and (6) promoting awareness of the program. 

Four activities and events are considered under the evaluation phase: <1) pro- 
giam validation, (2) settlement of the LSC contract, *3) program evaluation, and <4> 
the decision about the future of the program. 
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Part 1 



Introduction 



I. PURPOSE OF THE GUIDE 



OBJECTIVES 

Local educational agencies (LEAs) have entered into over 100 performance con- 
tracts with business firms and teacher groups since 1969. The essence of these 
contracts is that a learning system contractor (LSC) manages an instruction pro- 
gram for a specified group of students and is paid according to their measured 
achievement. 

Schools ordinarily provide for instructional programs by letting fixed contracts 
for resources. Teachers are hired on yearly contracts for fixed sums; textbooks are 
purchased at fixed prices per thousand copies; and similar contracts based on fixed 
amounts per unit are used to acquire other inputs. The school system then organizes 
these resources into an educational program and manages the program through its 
employees. An LSC, in contrast, is "paid for results” as measured by some index of 
student achievement. 

The LSCs in most programs have been commercial enterprises, but some have 
been individual teachers or organized teacher groups. Some observers believe that 
performance contracting will increasingly be of this "internal” type, between teach- 
ers and LEAs, rather than the "external” type between LEAs and outside enter- 
prises. 1 The two contract types share many features, but external contracts entail 
some special problems that are not present in contracts between an LEA and its 
employees. Eor that reason, and because commercial contracts have been by far the 
more common, this Guide concentrates largely on contracts between school districts 
and commercial firms. 

"Paying for results” sounds like a simple concept but in practice it has many 
ramifications, in light of which particular LEAs may find good reason for shunning 
performance contracts. Consequently, this Guide is not intended as an endorsement 
or sales brochure for performance contracting. Its objective is to present briefly the 
major planning, management, and evaluation issues associated with this technique. 

1 Internal programs were conducted during 1969-70 in Portland, Oregon, and in Stockton, California 
and Mesa, Arizona during 1970-71. Dade County (Miami), Florida, entered two internal and two external 
contracts during 1971-72. 
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Our hope is that this information will assist decisionmakers who are contemplating 
performance contracting programs, and that the Guide will help LEAs to structure 
the programs they do undertake. 



SPECIAL REQUIREMENTS 

Like any other new or special program, performance contracting levies special 
resource and skill requirements on an LEA. This Guide does not dwell on the 
features common to all innovations, but focuses on three areas where performance 
contracting levies special requirements: 

* Legal research and possibly legal action 

* Program management 

* Measurement and validation of student learning 

All school districts contract for materials, equipment, and personnel. Perform- 
ance contracts, however, open up a poorly charted area of contract and school law, 
requiring special legal attention. Research and possibly action may be required to 
resolve questions. Although variations among contracts and among state and local 
statutes make it hard to generalize about legal issues. Sec. Ill is devoted to some 
basic topics that merit consideration. 

An LEA considering a performance contracting program must be prepared to 
provide effective program management. Any innovative program requires good 
management, but it is particularly important when the LEA must direct a contrac- 
tor’s activities. 

In the initial phases of a program the LEA must: 

* Assess the educational needs to be filled 

* Write the Request for Proposal (RFP) or similar document 

* Select the contractor 

* Write and negotiate the contract 

LEA management must be all the more effective when a program is under way, 
especially if it uses materials, techniques, or personnel new to the district. Many 
questions will arise; only a dedicated and able manager will be able to resolve them 
without excessive loss of time or educational effectiveness. Various sections of this 
Guide are devoted to these special managerial requirements. 

The essence of performance contracting is the measurement of student achieve- 
ment. This feature makes a sound evaluation essential. With the contractor’s pay- 
ment hinging on student learning, measurement must be both reliable and valid. In 
turn, accuracy and objectivity on the part of the evaluator become of extreme 
importance. 

Validation is an audit of the appropriateness of the program’s evaluation proce- 



dures and verification of the results as reported by the project evaluation.* Ideally, 
it should include a review and critique of the criteria for test selection, an assess- 
ment of the reliability and validity of the tests as measures of the student learning 
desired, inspection of the conditions of test administration, checks of test-scoring 
procedures, and checks of procedures in computation for payment. Only in a few 
programs to date have these tasks been accomplished (Texarkana is a noteworthy 
example). Lacking such careful assessment, many LEAs and LSCs have felt that the 
means for determining contractor payment were not wholly satisfactory. A well- 
planned and well-executed validation of student achievement therefore should be an 
integral part of any performance contracting program. This Guide devotes several 
sections to validation and evaluation issues, and the Technical Appendix goes into 
even more detail. 



OUTLINE OF THE GUIDE 

The Guide is divided into four major parts. Part 1 is an introductory section 
discussing the performance contracting concept, some past applications, and the 
major requirements a program generates for an LEA. Part 2 discusses the planning 
and contracting phases of a program. Part 3 considers the operational phase. Part 
4 discusses evaluation. The individual sections reflect the sequence of major events 
in a typical program as depicted in Fig. 1. 

2 Robert E. Kraner, Final Audit Report. Texarkana Dropout Prevention Program . EPIC Diversified 
Systems. Tucson. Arizona, 1970. 
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II. THE PERFORMANCE CONTRACTING CONCEPT 



SOME BASIC QUESTIONS 

Six questions are often asked about performance contracting: 

• Is it legal? 

• Will it produce results? 

• Can learning be purchased? 

• Where has it been tried? 

• What kinds of programs do contractors offer? 

• What do performance contracting programs cost? 



IS PERFORMANCE CONTRACTING LEGAL? 

There is no simple yes or no answer. The legal status of performance contract- 
ing programs is unclear and in many jurisdictions it is changing. The structure of 
any particular program is also a factor. Section III will discuss these consideration^. 
Despite some ambiguity, however, programs continue to be implemented in many 
states. 



WILL PERFORMANCE CONTRACTING PRODUCE RESULTS? 

Three goals have predominated in past programs: 

• A substantial improvement in compensatory education 

• Facilitating instructional innovations 

• Development of a system of educational accountability 

It is hard to generalize about performance contracting because of substantial 
differences among program characteristics and outcomes. A Rand study of 197071 



programs, however, reached the following conclusions/ 5 

First, although some programs recorded significant improvements in achieve- 
ment scores, the gains in most programs were not high enough or uniform enough 
to indicate that performance contracting is a sure solution to compensatory educa- 
tion problems." 1 

Second, performance contracting seems to be an effective change agent. Per- 
formance contracting programs have been successful in introducing new materials 
and methods into the educational process. 

Third, performance contracting programs have given LEAs the opportunity to 
explore educational accountability. Such programs virtually demand clear defini- 
tions of educational goals, and have pushed the development of techniques for 
analyzing achievement results. 



CAN LEARNING BE PURCHASED? 

In a very literal sense, schools have always purchased learning. They hire 
teachers, buy materials, and assemble other resources, knowing from long experi- 
ence that some amount of learning will result. 

No one. however, can "guarantee" that another person will learn. The linking 
of payments to achievement scores, <vhich is the essence of performance contracting, 
is far different from "guaranteeing" that the program will achieve the objectives 
that the LEA seeks. 

This last point has a vital implication for the question of the success of pro- 
grams. It is common to compare observed achievement gains against the goals 
expressed in the contract. However, it is more meaningful to compare the gains 
realized by the same students in prior years, or the gains realized bv some relevant 
comparison group such as students in other remedial programs in the district, or to 
assess the contribution ofthe performance contracting program to the LEA s overall 
program. 



WHERE HAS IT BEEN TRIED? 

There have been performance contracts in many and diverse school districts, as 
Table 1 reveals. Lengthy as it is, the list in Table 1 is not exhaustive, since it excludes 
many programs that have not received publicity. 

3 Polly Carpenter <*nd George R. Hall. Case Studies in Educational Performance Contracting: I. 
Conclusions and Implications. The Rand Corporation. R-900/1-HEW, December 1971. 

1 The OEO experience was even more discouraging. In general the contractors were no more success- 
ful than conventional classrooms in improving reading and mathematics skills. An Experiment in Per- 
formance Contracting: Summary of Preliminary Results, OEO Pamphlet 3400-5. Office of Economic Oppor- 
tunity. February 1972. p. 17. 
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PERFORMANCE CONTRACTING PROGRAMS FOR 19 7U- 7 1 








* LEA was contractor for 0K0. CSC hail Biihcont ract with 1.KA. 

State Department of education did Initial planning; contract was between ISC and UG\ 
L Targct payment was $B0 for all seven Virginia programs. 

Target payment was $167 for all three Colorado programs. 
t l , lus regular salaries. 



WHAT KINDS OF PROGRAMS DO CONTRACTORS OFFER? 

While there has been some experimentation with vocational education pro- 
grams/’’ most programs have involved reading, often together with mathematics, 
partly because most have been remedial: compensatory education focuses on basic 
skills. This concentration also reflects the necessity for being able to measure re- 
sults. Theoretically, any subject could be taught under a performance contract with 
the payment related to some index of outcome; but measurement techniques for 
reading and mathematics (norm-referenced achievement tests, which still leave 
much to be desired as measures of program success) are far ahead of those for other 
subjects. 

Vocational education courses are amenable to performance contracting since 
craft and unior standards provide accepted definitions of achievement; even here, 
however, the administration of tests of skills is expensive, and unions often resort 
to measures of training input <such as ”a year of instruction”) rather than direct 
measures of skill. Be that as it may, performance contracting is more likely to 
expand in vocational than in academic subjects. 

In short, performance contracting is feasible ifand only if both parties can agree 
on how to measure the contractor’s work quantitatively. For now and the near 
future this means that contracting programs will focus on basic and vocational 
skills. 

Most programs have operated as components within a conventional school set- 
ting. If a contractor is to teach reading to 180 students in three grades, let us say, 
he is assigned a classroom and the students come to him in groups of 30 for an hour s 
instruction. For the rest of the day they follow the ordinary school schedule. 

There are some notable exceptions to this pattern. Behavioral Research 
Laboratories, for example, is providing the entire curriculum for one elementary 
school in Gavy, Indiana. The payment to BRL. however, is based on the students’ 
achievement only in reading and mathematics. 

Performance contracting programs have differed so widely that it is hard to 
speak of a typical program. In general, the contractors have used new materials and 
teaching techniques with special emphasis on individualized diagnoses and specifica- 
tions of instruction. A few contractors have used educational hardware extensively. 
Most have introduced changes in classroom management techniques — notably, us- 
ing the certificated teacher more as an organizer of instruction and an analyst of 
individual students’ needs, and less as a direct transmitter of instruction. A number 
of programs have used paraprofessionals. 

Some, but not all, contractors have emphasized changing the classroom envi- 
ronment by carpeting floors, setting up "reinforcement centers,” substituting car- 
rels for conventional desks and tables, and the like. 

In Dallas. Texas, some vocational skills have been taught under contract, and the Dallas school 
system is expanding the program. Muskegon. Michigan, has also been developing a vocational education 
program and other LEAs have shown interest in doing the same. See D. R. Waldrip. "Performance 
Contracting: The Dallas Experiment." unpublished paper. Dallas Independent School District, Dallas. 
Texas, n .d. 
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Some contractors have used contingency management techniques and special 
incentives for children. A few programs have also offered special bonuses to teach- 
ers, but most programs have provided conventional rewards to students and teach- 
ers. 

Programs have differed in the extent of LSC involvement. In most, the contrac- 
tor has been the prime manager of the learning program; in other programs, such 
as those under OEO sponsorship, the LEA has retained the authority to approve all 
aspects of program changes, though the contractors have had considerable authority 
over day-to-day operations. In a few programs (e.g., Philadelphia, Portland, Oregon, 
and Flint, Michigan) the LEA retained almost complete operating authority: the 
contractors did no more than supply educational materials and train the districts 
teachers in using them. The price of the materials was contingent on the achie\e- 
ment of the students in the programs. 

Most contractors have been directly involved in the classroom, but the teachers 
have usually been LEA employees. Most contractors have viewed their classroom 
activities as a p as sing phase, leading to "turnkeyed” systems whereby the LEAs 
ultimatelv take over and operate the new systems as part of their regular programs. 

Many' programs have been financed out of various categorical funds such as the 
Elementary and Secondary Education Act or other special Federal and state assist- 
ance funds”. The OEO programs were financed out of OEO’s research and develop- 
ment funds. A number of programs have been financed wholly or in part by regular 
school district funds. 

In sum, any LEA considering a performance contracting program has a wide 
choice among programs. There is no "model” performance contracting program. A 
host of different arrangements have been used. LEAs therefore have to evaluate the 
usefulness of relating payment for instructional services to student learning, and 
also determine which variant, if any, is most appropriate. 



WHAT IS THE COST? 

Table 1 has shown some figures for target prices, but they should not be con- 
fused with costs. First, the programs have differed widely as to the expenses shown 
on the contractor’s books and those paid directly by the school district. Teacher 
salaries are the prime example, but clerical, evaluation, and many other expenses 
also have been accounted for differently from program to program. 

Second, it is improper to make interdistrict comparisons of payments, because 
salaries and other prices differ widely among districts. Third, the figures shown in 
Table 1 are target prices that would have been paid if the achievement-gain goals 
had been realized. Since actual results have differed from targets, actual payments 
would differ from the figures shown in Table 1. Very few actual prices have been 
made public, however. 

In general, the cost-per-student (on a comparable replication cost basis) in per- 
formance contracting programs will be about the same as or somewhat less than 




that in purely remedial programs/ 1 As is true in most compensatory education 
programs, however, this ''ost is higher than it is for conventional programs. 

Performance contracting programs have usually had a different resource struc- 
ture from that in conventional remedial programs. Most contracting programs have 
featured relatively high student-teacher ratios compared with the usual compensa- 
tory program, but have made extensive use of materials, paraprofessionals, class- 
room modifications, and sometimes machines. 

In sum, although performance contracting is generally mor? expensive than 
instruction in the conventional classroom, it remains financially attractive com- 
pared with other types of remedial education. 



See Carpenter and Hall. op. cit., for a discussion of program costs: and see the Technical ADpendix 
lor a discussion of resource and cost analysis in performance contracting programs, particularly the 
concept of comparable replication costs. 





III. LEGAL RESEARCH AND ACTION 



LEGAL RESEARCH 

Many of the legal implications of performance contracting remain cloudy. Until 
experience clarifies matters, school districts undertaking contract programs will 
need to engage in research and possibly action, in light of relevant state and local 
statutes. Specifically, the LEA should investigate the legalities touching on the 



• LEA authority to enter a performance contract 

• Authority over the project 

• Contracting 

• Instructional time-allotments 

• Textbooks and materials 

• Teacher qualifications and conditions of employment 

• Provision for student rights, responsibilities, and liability 

• Warranties, bonds and other legal protections 



AUTHORITY TO ENTER A PERFORMANCE CONTRACT 

An LEA may find that it lacks authority to enter into a performance contract, 
or authority may depend upon the source of funds. For example, the New York State 
Department of Education has decided against use of state money for performance 
contracting, but a New York school can still enter into such a program using Federal 
funds. 7 

The legal authority to enter a contract may depend on hew the program goals 
are designated. Three ways that have been used in the past to assure this authority 
are to designate the program (1) as an experiment, (2) as a program for disadvan- 
taged children, (3) as a phase in a "turnkey” program. 

• ’‘Performance Contracting in New York State.” Phi Delta Kappan . Vol. 52. No. 5. January- 1971. 



following: 




p. 323. 




Experimental programs frequently come under rules different from those for 
regular programs. For example, state textbook requirements and teacher certifica- 
tion rules may not apply. Of course, if the experimental program becomes a regular 
program, problems might then arise, but many districts have preferred to postpone 
such questions. 

In California, and possibly in other states, the Education Code in its section on 
the instruction of disadvantaged children specifically mentioned "private agency or 
organization.” Before 1972, it would appear that performance contracts for experi- 
mental compensatory programs would be authorized in California, but programs for 
other children would not. Recent legislation, however, appears to have removed this 
distinction in California. 

A third way LEAs have sought to assure authority to contract has been to 
designate the performance contract us the first stage in a "turnkey” program de- 
signed to yield new materials and procedures for in-house district use. But if an LEA 
commits itself in advance to transferring the program to intradistrict use, then 
performance contracting is weakened as a device for promoting accountability. 
Furthermore, such a posture may itself create legal problems, because it might be 
interpreted as premature approval of the outcome of the program. 

There are advantages to programs lasting more than a single year, but legal 
counsel for some LEAs have advised that contracts for more than one year would 
be illegal. On the other hand, Gary, Texarkana, and Cajon Valley, California, have 
been able to structure multiyear programs within the context of their respective 
state laws. 

If an LEA enters a contract without appropriate authority, the state may with- 
hold credit from students for the subjects taught by the LSC, or withhold funds from 
the school, or even decertify the school. But authority may not be clear and may 
depend upon other features nf vhe program such as the materials used, credentials 
of the contractor’s personnel., and the like. To date, each LEA considering a perform- 
ance contracting program has had to determine its own legal position; until legal 
principles become clarified they will probably have to continue to do so. 



AUTHORITY OVER THE PROJECT 

No LEA can legally waive responsibility for any program for its students. Any 
contract must clearly reserve ultimate control for the LEA. The central issue is how 
to preserve control and at the same time allow the contractor the necessary freedom 
to conduct his program. 

Three basic principles have been suggested: 8 



H Ibid. 



Safeguards should ensure that any "nontraditional” procedures will not 
harm children 

A principal or other official should have the power to intervene if students 



are being damaged 

Clear lines of authority and responsibility should be established among 
parents, school boards, principals* teachers, and contractor personnel. 



CONTRACTING REQUIREMENTS 

In a number of states contracts in excess of certain amounts must be advertised 
and let by means of competitive bids . 0 Sometimes there are exceptions to such rules 
that might cover a specific program. Therefore, an LEA should check the applicable 
contracting and procurement regulations. 



TIME ALLOCATIONS 

Most states have requirements or suggested standards on how much of th^ 
school day will be devoted to each of various required subjects. Several programs 
have run into problems with these standards, as for example in Gary, where the LSC 
manages Banneker School’s entire curriculum. Programs involving only one or two 
subjects and a fraction of the student’s day are less likely to face trouble. 

The LEA faces a different problem if state standards are suggestive rather than 
mandatory. In this case it may be possible to rearrange schedules to meet both 
program requirements and legal time-allocation standards. 



TEXTBOOKS 

Most states specify permissible textbooks and materials. An innovative pro- 
gram is likely to use books and materials not on the approved list. This problem has 
arisen in several program, and the most frequent solution has been to obtain waivers 
on the grounds that the programs are experimental. Other possible solutions would 
be to get Lhe materials approved or seek legislative change in the textbook laws. 



TEACHER QUALIFICATIONS AND CONDITIONS OF EMPLOYMENT 

The LEA should ensure that teachers assigned to a contracted program meet 
state requirements governing teachers’ credentials and qualifications. In most but 



9 Indiana, for example, requires competitive bids for purchases of materials, equipment, goods, and 
supplies exceeding $2000. Personal services contracts are exempt, but the Indiana State Department of 
Education criticized the Gary contract on the grounds that it involved more than services and therefore 
should not have been let sole-source. 



not all programs the LSC has used local teachers, a practice that usually avoids 
challenges regarding credentials but leaves open the question of whether the creden- 
tials are appropriate for the program. 

Problems are more likely to arise if the LSC brings in teachers or supervisors 
from outside the district. The LEA should make sure in advance that any personnel 
the LSC intends to assign to the program can meet certification requirements or that 
waivers can be obtained. 

If teachers operate under a collective bargaining agreement, teacher arrange- 
ments in the program will have to be consistent with union-agreement provisions, 
required employment practices, and state codes. (Many of the following remarks 
about unions also apply to LEAs where there are established teacher employment 
rights but no collective bargaining agreement.) Issues have arisen in five areas: 

© Union involvement in the planning process 

• Initial assignments and transfers of teachers 

• Subsequent transfers and reassign ments 

• Class sizes 

• Teacher compensation 

Collective bargaining agreements usually contain a clause stipulating that all 
changes in pay or conditions of work will be the subject of LEA and teacher-group 
negotiations. In some cities teachers’ unions have claimed the right to participate 
in the initial planning for any performance contract program, arguing that it would 
affect working conditions. In the cases we know about, the LEA did not accede to 
such claims. Claims are likely to arise again, however. 

The initial selection of teachers may involve assignments and transfers that 
raise questions about seniority rights. In the past, some LEAs have argued that 
performance contracting programs create special requirements and therefore super- 
sede normal transfer provisions. 

Many performance contracts have contained clauses permitting the contractor 
to reassign a teacher on short notice if his work is unsatisfactory. The advantage to 
the LSC of such an arrangement is obvious, but it may not accord with standard 
district employment practice. 

Class size may become an issue. Classroom instruction in contracting programs 
has often been done by a team composed of a certificated teacher and a paraprofes- 
sional, making extensive use of materials. The pupil-to-adult ratio has frequently 
been low, but the pupil-to-certificated teacher ratio likely will be higher than in 
conventional classrooms (and a few programs have operated with no certificated 
teachers). This may lead to conflicts. 

Teacher compensation can become an issue, particularly if the contractor pro- 
poses to pay bonuses to teachers on the basis of student achievement. At the mini- 
mum a union is likely to insist that all teachers in the district receive bonuses 
similar to those received by the teachers in the program (this happened in one school 
district, which immediately insisted the LSC withdraw the bonuses). 

The usual arrangement is that teachers in performance contracting programs 



are paid according to the standard LEA pay scale. In any event, it is important to 
make sure that all parties understand and agree on the pay arrangements, to avoid 
conflicts between the LEA-LSC contract and the LEA-union contract. 



RESPONSIBILITIES AND LIABILITIES REGARDING STUDENTS 

The LEA should ensure in advance that its responsibilities regarding students 
will be discharged under the contract, that the LSC understands its responsibilities, 
and that all rights and liabilities have been specified. 



WARRANTIES, BONDS, AND OTHER LEGAL PROTECTION 

Some contracts have included clauses requiring tftat the contractor post per- 
formance bonds, guarantee prices, or maintain fixed prices for any materials or 
equipment purchased. Many of tine clauses have followed commercial contracting 
practice, but it may be counterproductive to include in a performance contract all 
the protective clauses commonly included in contracts for standardized materials 
such as furniture or motion picture projectors. Because a performance contracting 
program is an attempt to deve-op innovative procedures, uncertainties and changes 
are inevitable. In such cases, a contract bristling with constraints can cause more 
trouble than it is worth. In short, the standard protective clauses should probably 
be included, but the LEA should avoid "overengineering the agreement. 10 



LEGAL ACTION 

Research mey be insufficient to resolve all questions, and legal action may be 
required. In some programs the contract and planning documents have been submit- 
ted to the LEA’s state attorney general or department of education for an advisory 
opinion. Doing so means the LEA will know where its program stands with state 
officials, bvu, if there is an adverse opinion it may be difficult to proceed even if the 
LEA feels the state’s views are debatable. 

Despite much talk about judicial challenges, few if any suits have yet been 
taken to court. In past years critics of performance contracting programs have been 
persuaded to postpone formal challenges on the basis that the programs were experi- 
mental. T n future years, this argument is unlikely to be as persuasive. 

Various state legislatures have considered or are considering statutes involving 
the status of performance contracting. As performance contracting becomes less of 
a novelty, LEAs should be prepared to devote more attention to legal matters. 

10 The Norfolk-LRA Agreement in the Technical Appendix is a good example of a contract with many 
protective clauses. Most, if not all, of the contingencies that arose in Norfo’k were covered by the contract. 



Part 2 



Planning and Contracting 



IV. PROGRAM DEFINITION 



PROGRAM GOALS 

The first step in planning the program is to assess needs and define goals and 
objectives. This need not be elaborate but it is essential for a coherent program. 

Probably there will be a hierarchy of goals and objectives. At the most general 
level the LEA may have one or more basic educational goals such as improved 
community support, racial integration, or a more modern curriculum. These must 
be translated into program objectives. Program objectives must be expressed in 
specific terms, such as lowering the student dropout rate, increasing the rate of 
cognitive growth, and improving student self-images and attitudes. Finally, the LEA 
must translate the program’s objectives into the specifications for contractor pay- 
ment, or contract objectives. Examples of such objectives might be doubling each 
student’s rate of cognitive-skill achievement as measured by a standardized test, 
having students qualify for on-the-job vocational training, or the attainment of some 
other measurable criterion of performance. 

To link payment to results, the LEA needs to specify appropriate measures. 
Measures do not necessarily have to be limited to a norm-referenced test. Criterion- 
referenced tests of behavioral objectives have been used along with norm-referenced 
tests in some programs. Quantitative measures of achievement such as the student 
dropout rate might also be used if, for example, lowering the dropout rate were a 
program objective. The reliability and validity of the proposed measures, however, 
are basic questions; at the start of a program the LEA should assure itself that the 
measures are accurate and relevant for the program objectives. If objectives cannot 
be specified or if there are no meaningful quantitative measures, a performance 
contract is inappropriate. 

There is a natural tendency for an LEA to detail a large number of program 
objectives, but doing so will make it difficult to find an LSC who can accept them and 
even more so to find one that will be able to meet them all. 11 Most LSCs have a fixed 

11 The experience of Yuba County. California is an apt example of thes^ problems. Yuba wanted a 
program that would increase the "positive attitude” of mathematics teachers. The contractor s payment 
would be based, the LEA. ‘ecided, primarily on achievement of two objectives: improvement in student 
scores on standardized norm-referenced tests, and improved teacher attitudes. Teacher improvement -was 
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repertoire of techniques and materials designed to attain a finite set of educational 
objectives. It is unrealistic to expect them to meet a wide array of targets. 



USE OF SUPPORT CONTRACTORS 

A basic decision the LEA must make is whether to use support contractors, and 
in what roles. Many LEAs have provided their own management, evaluation, and 
auditing, but many others have employed business firms, governmental units, or 
university groups. The schematic outline of the program has been expanded in Fig. 
2 to indicate some of the optional roles support contractors can play. 

If a management support group (MSG) is employed, it may assist the LEA in one 
or more of such tasks as: needs assessment, RFP preparation, LSC selection, drawing 
up of the contract, program monitoring (e.g., collection of data on program cost and 
program implementation), evaluation monitoring, and consultation on the future of 
the program. 

An independent evaluator will usually assist in designing the evaluation, select- 
ing or designing specific measuring instruments, administering tests, scoring tests, 
and computing contractor payment. An independent evaluation, among other con- 
tributions, enhances the credibility of program outcomes. 

Educational auditors, in programs where they have been used, have analyzed 
evaluation designs, observed test administration, and performed other validating 
functions. The use of an auditor to validate measures of educational outcome offers 
many advantages. Notably, an auditor relieves the evaluator of the onus for guaran- 
teeing the validity of test results, and frees him to focus on program improvement. 

Against the arguments for the use of support contractors must be posed the 
advantages to the district of performing these functions "in-house.” Because good 
management support contractors and evaluators are expensive, some LEAs feel it 
is more economical to perform these functions themselves. Another, and more im- 
pressive, consideration cited by some LEAs is that without supoort contractors the 
LEAs can preserve tighter cognizance and control. The more agencies involved, the 
harder it is to administer the program and maintain coordination. Also, by perform- 
ing the support functions itself, the LEA gains valuable experience that can be 
2 PPbed in other performance contracting programs and educational programs gen- 
erally, assuming the LEA has the necessary expertise and personnel available. 

If the LEA decides to use support contractors, then it alone should select the 
contractors. The Management Support Group (MSG), evaluator, or auditor should 
be financially responsible only to tne LEA. There should be no direct contractual 
link between the LSC and any support contractors. In some past programs there has 
been joint selection by the LEA and LSD, and in one program the evaluator was paid 
by the LSC, which led to the evaluator’s objectivity being questioned. 

to be measured by a system to be designed by the Yuba County Schools. The Request for Proposal was 
circulated to 'IS institutions, agencies, and private companies but no proposals were returned. The 
potential respondents were concerned about the highly specified program, the intangible natu.e of the 
attitudinal goal, and the uncertainty about how achievement would be measured. 



PLANNING & CONTRACTING OPERATION EVALUATION 




Fig. 2— Performance contracting sequence 



TEACHER INVOLVEMENT 



An LEA should make plans and carry them through to involve teachers from 
the start in the planning and implementation of a contracting program. This 
action may produce two benefits: 

• It may elicit information concerning the educational needs of students. 

• It may promote the active support of teachers and other district em- 
ployees. which is important if the performance contract is to solve rather 
than create problems. 

Most LSCs have employed teachers already in the school. This practice makes 
a program less ‘’threatening*’ and also avoids a new teacher's having to learn the 
local ground rules. It also taps the local teachers* knowledge of’ the students and 
their needs and abilities . 1 * 1 

Even if personnel come from the existing LEA staff, teachers are likely to 
regard performance contracting as something of a threat. The American Federation 
of Teachers and the National Education Association have been highly critical, partly 
because most programs combine the use of paraprofessionals and materials with 
larger class sizes — features not likely to promote teacher enthusiasm. The program 
may create some scheduling problems for the host school and its teachers: but more 
generally, teachers are not Tattered by school boards inviting "outsiders” to conduct 
educational programs. Nonetheless, many teachers have supported past perform- 
ance contracting programs — they have even acted as LSCs in some school districts. 
Teachers are interested in new skills, techniques, and materials, and they are con- 
cerned about compensatory^ education problems. A program that gives them the 
opportunity to try something new and increase their abilities is likely to be attrac- 
tive. 



Ia For more on this point see Carpenter and Hall. op. cit.. pp. 33-34. 

1;t Sometimes the teachers have been carried on the L-SC’s payroll: often they have remained on t ie 
LEA*s payroll to simplify tenure and fringe-benefit arrangements. 



V. PLANNING THE EVALUATION 



INTRODUCTION 

Evaluation is commonly regarded as a postprogram function, but it is vital that 
it be planned for early, when program operations are planned. If LEA officials are 
to judge the effectiveness of a program, and make rational decisions about its con- 
tinuance, they must have a sound analysis of its outcomes- 

To assure that decision-relevant information is obtained, a thorough but flexible 
evaluation design must be planned before the start of the program . 14 Without sub- 
stantial evaluation planning there is a high probability c; poorly selected or de- 
signed measuring instruments, untested students, poor test conditions, and incom- 
plete or irrelevant criteria. 

Designing an evaluation is an intellectual challenge. These programs are quasi- 
experiments or demonstrations rather than controlled laboratory experiments. A 
school cannot and should not maintain the same type of scientific controls over the 
environment that can be maintained in a laboratory. The difficulty is that the 
standard evaluation methods were designed with laboratories in mind; evaluation 
techniques for quasi-experiments are much less developed. 1 '* This lack makes it all 
the more important for an LEA to have a flexible, thorough, and timely evaluation 
plan. 

There are four basic steps in developing a plan: 

• Determining the functions of the evaluation 

• Selecting the evaluator 

• Selecting the evaluation criteria 

• Planning the testing program 

Flexibility is important. Without it the evaluator can be. as in the 1969-70 Texarkana program, 
in the position of having planned for a program that was substantially revised, and therefore having to 
choose between evaluating a planned program that was not implemented or evaluating a program 
without a relevant evaluation plan. See P- Carpenter. A. W. Cha.fant, and G. R. Hall. Case Studies in 
Educational Performance Contracting: 3. Texarkana . Arkansas. Li** -ty-Eyla u. Texas. The Rand Corpora- 
tion. R-900/3-HEW, December 1971. pp. 21-23. 

l% The Bibliography provides references to the literature on evaluation oi quasi-experiments. 
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DETERMINING THE FUNCTIONS OF THE EVALUATION 

The evaluation may be assigned one or more of four functions: 

• Measuring outcomes for contract settlement 

• Validation (audit) of the measures of program outcomes 

• Helping decisionmakers to decide about the future of the program 

• Assisting in program improvement 

Someone has to perform all four functions, and it is tempting to assign them all to 
the evaluator. Because of conflicts among these functions, however, the LEA may 
want to divide them among the program participants. 

If the evaluator is responsible not only for measuring achievement but also for 
certifying that the measurements are valid and reliable, he is in the position of 
having to validate his own work. If he accepts such a role, he may well be reluctant 
to discuss program and evaluation problems with teachers or other program person- 
nel. 

The evaluator’s most valuable contribution may be the provision of data to LEA 
officials who have to decide w'hether to continue the program. If done conscien- 
tiously, this task will require collection of more data of more kinds than are required 
to settle contracts. The LEA will want to know, among other things, about costs, 
reactions of teachers and parents, and various scholastic and affective influences on 
students. If the evaluator is assigned this sort of decision-support role, either he will 
have to be provided with more resources than he would otherwise receive, or he will 
have to devote less time to measuring student achievement. 

Finally, the evaluator can provide on-site monitoring that can help operating 
personnel improve their programs as they go. Again, however, he would require 
more resources than he would need for contract settlement alone. Also, if he is 
responsible for certifying the validity of the measurements he may be unwilling to 
provide much feedback to program personnel, for fear that feedback might instigate 
test-teaching. Moreover, if the evaluator is also concerned about contract settlement 
he may be unwilling to release data until both the LEA and LSC have agreed on the 
final settlement. Thus, in contrast with other programs, evaluators may be much 
less willing to provide "Veal time’" data during performance contracting programs. 

The important point for the LEA's plans is that all four functions have to be 
performed. If more than one is assigned to the evaluator, the LEA should think 
about possible conflicts. If only one is assigned, the LEA must consider what part ies 
will perform the other functions, and what their relationships will be. 



SELECTING THE EVALUATOR 

The choice of an evaluator depends on his assigned functions. If he only has to 
measure or validate program outcome, a multitude of universities, nonprofit groups, 
educational agencies, and business firms have the required resources and skills. It 
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is intuitively obvious that the LSC should not perform these functions. In some p^st 
programs the LEA has provided the achievement measurements and/or validation. 
Using an independent evaluator or auditor is preferable, however, because critics 
may argue that the LEA has an interest in a favorable assessment of the program. 

If the evaluator is to provide information for LEA decisions, then a new set of 
qualifications arises. The LEA will want an organization that can work effectively 
with the public officials involved and that has the required skills in resource and 
educational analysis. The LEA may decide to assign this function to an internal 
group or to an independent management support contractor rather than an evalua- 
tor. If so, however, the division of responsibility among the various groups assessing 
program outcomes should be made clear. 

Evaluation for program improvement levies the largest demands for staff skills 
and resources because it requires frequent visits for observation, interviews, and 
data collection. The evaluator must establish rapport with program personnel and 
win their trust, while maintaining objectivity about the program s results- Both 
internal LEA groups and external organizations have been used in this function in 
past programs. The choice will probably depend upon whether the LEA has the 
necessary people with the required skills. 



SELECTING EV ALUATION CRITERIA 

The important task in evaluation planning is determining how to define and 
measure program success. If settling the contract is the chief aim, the measurement 
problem is fairly simple. Success in this context can be determined by comparing 
actual achievement measurements with whatever goals were established in the 
contract. 

Much publicity has been given to such computations. At best, however, they are 
incomplete and can often be misleading. Any goal, large or small, can be specified 
in a contract. A more meaningful test of program success than meeting achieve- 
ment-test targets may be whether it was more cost-effective than other potential 
programs or, more generally*, whether it made a net contribution to the LEAs 
overall program. 

To date most measures of program success have been limited to standardized 
norm-referenced tests and, to a lesser extent, criterion-referenced tests. The usual 
standardized tests were not designed to measure the effectiveness, for individual 
students, of shorVterm instructional programs; consequently, a number of statistical 
and practical problems beset the use of such tests in this mode. The literature on 
this issue is reviewed in Technical Appendix A, R-955/2-HEW. 16 Despite their draw- 
backs, standardized norm-referenced student achievement tests will likely* continue 
to be used for want of a better measure. Their imperfections, and the suggestions 
;n the Technical Appendix for limiting their impact, should be kept in mind. In 

q ir * Prepared bv Dr. J. Richard Harsh of Educational Testing Service. 

ERIC 
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particular, performance contracts should cover a long enough time period to make 
it likely that gain-differences will represent real achievement gains and not short- 
term statistical errors. 

Criterion-referenced or learning mastery tests at first glance appear to be an 
attractive altern."-: „e to norm-referenced tests. Student mastery is judged by cri- 
teria associated the educational objective itself rather than by the student’s 
standing with res^ to his peers. A review of experience in past programs, how- 
ever, indicates that severe problems afflict the use of such tests . 17 

* The needed tests probably do not exist; and if they have to be devised after 
the program is started, it is likely that testing schedules will not be met. 

* Few if any instruments have been field tested for reliability. 

♦ Criterion-referenced teste pose significant difficulties in test administra- 
tion. 

— Some require special training of test administrators. 

— Since they are tied to individualized curricula, the evaluator must use 
sampling techniques for students and objectives to keep required re- 
sources within reasonable bounds. 

• Unless they are given on a pre- and post-test basis, it may be open to 
question whether the students had already mastered the objectives before 
entering the program. 

Few programs have gone beyond norm-referenced or criterion-referenced test 
results for payment purposes. There is vddespread agreement, however, that for 
determining the future of a program other data are important. Some LEAs have 
considered using other measures of program outcome for contract payment pur- 
poses, but have been discouraged in their attempts to get quantitative data that are 
reliable and valid. Even if the LEA cannot link such outcomes to contractor pay- 
ments, however, it is still important that the LEA plan to collect data other than 
test scores for management and evaluation. 

A rough achievement measure can be obtained by sampling the classroom work 
of program students and of regular students with comparable academic ability at 
two or more points in time . 18 Ranking techniques can be used to carry out such an 
analysis . 19 The results may also suggest the program’s effects on student attitudes, 
which may be even more important than achievement but which are gains difficult 



17 The problems were dramatically illustrated in the OEO performance contracting experiment. It 
had been planned that 25 percent of the contractors’ pay would be based on criterion referenced examina- 
tions that OEO called interim performance objective tests (IPOs). The problem of the large number of 
tests and test items required by the individualized curriculum and the changes in curricula during the 
school year made it unrealistic for contractors to submit tests in advance to the evaluator for analysis 
and approval. The result was, according to OEO, that some of the tests were too easy and others failed 
to measure what contractors had taught. The OEO report concluded that, . . the IPOs appear to have 
been virtually useless for evaluation purposes and to have had questionable value for payment purposes. 
Office of Economic Opportunity, op. cit., pp. 15-16. See also Appendix A of the Technical Appendix. 

lK See, for example, W. S. Torgerson, Theory and Methods of Scaling, John Wiley and Sons, New V ork, 
195S. 

1:1 T. S. Donaldson, Subjective Scaling of Student Performance . The Rand Corporation, P-4596. March 
1971. 
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to measure in this way because there is no simple and direct relationship to achieve- 
ment gains. Evaiuators often use survey questionnaires to assess attitude changes, 
but these generally have low validity and sometimes low reliability. It is possible to 
assess attitudes fairly reliably, however, through structured interviews by trained 
psychometrisi?. Ideally, such interviews should be validated by direct observation 
of student behavior in appropriate situations. Another approach is to try to infer 
affective changes from behavioral data such as vandalism rates, dropout rates, police 
actions, disciplinary problems, attendance rates, and the like. The task is easier at 
the high school level, where such variables can reasonably be regarded as reflecting 
student attitudes toward school. At the grade school level, attendance seems to be 
the best index. Referrals to the principals for discipline are also a relevant measure 
but they are highly dependent on teachers’ tolerances for deviant behavior. 

Affective results can be measured and can provide a good basis for program 
decisions. The 1970-71 Texarkana program is a case in point. The evaluator moni- 
tored dropout rates as well as cognitive achievement growth. This factor w'as espe- 
cially important since the basic program objective was dropout prevention. Despite 
low achievement gains on standardized achievement tests, the evaluator recom- 
mended, and the school system adopted, a policy of continuing the program without 
contractor participation primarily because of a sharp reduction in the dropout rate. 

Teacher attitudes toward performance contracting programs vary. In some pro- 
grams teachers have enjoyed the opportunity to learn new teaching techniques and 
explore new materials. In others some teachers have felt overburdened by new 
duties. The independent evaluations that we have seen have generally surveyed or 
interviewed teachers in the program and reported their reactions., but have less 
often examined reactions of teachers outside the program. In several of the pro- 
grams Rand has studied, outside teachers knew little about the way the programs 
operated. If turnkeying technology is a goal, teachers throughout the district should 
be fully informed about it. Reactions of administrators and union spokesmen are 
also relevant, and it is wise to assure that all factions are heard, not merely the most 
vocal. 

The influence of performance contracts on parental attitudes may turn out to 
be the most Important measure of success for some programs. Recorded test gains 
may be outstanding or no more than modest, but cold facts and figures do not carry 
the force of parents who believe that educational quality has been improved. The 
most important indicator is likely to be their participation in parent organizations 
that most performance contractors try to organize. If a substantial percentage of 
parents come out to meetings and thereby help to shape or monitor these programs, 
it is safe to assume there is genuine interest. 

Decisions about the future of the program must rest on a solid understanding 
of what "the program” really is. Certainly in the first year or two, performance 
contracting programs are developmental, altering as they go along to adjust to 
unforeseen problems and to take advantage of unexpected opportunities. While the 
end result may well be an increase in students' achievement scores, the program in 
its ultimate successful mutation may seem like a different species from the program 
described in the proposal or the contract. The evaluator should note major changes 
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in the instructional process, in the use of resources, in the teachers or students 
involved, or in the program direction. Otherwise, "the program” that is continued 
or expanded in the future may be an imperfect model of what actually went on. 

In developing an evaluation plan it is useful to designate control or comparison 
groups. Such comparison is not necessary for settling a contract but it is extremely 
useful for decisions about a program’s future. In some programs ihe history of the 
treatment group has been used as the control. If the school or program, however, 
has had a number of transfers it may be difficult to obtain the necessary historical 
data. It is usually easier to designate some other group of students for comparison 
purposes. 



PLANNING THE TESTING PROGRAM 

Any performance contracting program involves a great deal of testing for: 

• Diagnostic/prescriptive purposes 

• Determining mastery of assigned material 

• Determining payment to contractor 

• Evaluation purposes and for the LEA’s general test program 

But the more tests administered, the less time available for instruction and the 
greater the possibility of "overtesting” and thereby decreasing the interest of stu- 
dents in scoring well on tests. The original test program design should therefore be 
sparse; tests can be added as their desirability becomes apparent. 

The choice of the proper level of test difficulty may have to be deferred until 
students have been chosen for the program, and even then some students may be 
given tests at the wrong level of difficulty for them. This problem is extremely 
significant. If the test is too difficult for the student, his score may fall within the 
chance range and the meaning of any computation of achievement gain will be 
questionable, regardless of how well other aspects of the test program or of the 
performance contracting program itself are carried out. Such a situation is espe- 
cially likely to arise with standardized, norm-referenced achievement tests, because 
the tendency will be to give the test normed for the students’ nominal grade level. 
But a ninth-grade test given to a ninth-grade class full of students who are at least 
two grade levels behind will produce only frustration and confused data. A test that 
reflects their actual achievement levels will yield a more realistic picture — but 
trying to select such a test raises questions about how to interpret scores. It may be 
necessary to administer tests of several levels of difficulty if the spread of student 
capability is very large. For example, ninth-grade students may be reading at all 
levels from pre-primer to the seventh grade. For children in the lower grades who 
have reading difficulties, it may be well-nigh impossible to find a usable written test; 
some oral test may be required. 

There is another drawback to administering a test designed for a younger 
chronological age. A twelve-year-old may only have the reading skills of the average 



nine-year-old but he will be interested in different topics. If the subject matter is 
beneath him, his annoyance may damage his motivation to do his best. 

Selection of appropriate tests of performance objectives, if these are being used 
to determine contractor payment, poses a special problem. Their validity depends 
upon the accuracy of the diagnostic pretests, which pinpoint performance objectives 
for each student. If these tests yield the correct set of objectives, a high score on the 
post-test indicates that the student mastered the objectives assigned to him and is, 
therefore, a measure of achievement. An invalid diagnostic test may generate per- 
formance objectives that are too low, in which case a high post-test score would not 
necessarily indicate real achievement. In short, if a student fails a post-test there 
is no way to know, unless one is sure the diagnostic test was valid, whether the 
instruction was inadequate or the objectives were beyond his level of capability. If 
the student passes the test, there is no way to know whether the instruction was 
effective or the objectives were easy for him. 

If the contractor has already validated a complete diagnostic/prescriptive pro- 
gram of tests and performance objectives for the target population, this problem 
may not arise, but considering the current state of the art in criterion reference 
testing, it probably will arise. More likely, the ability of he classroom teacher will 
determine whether appropriate objectives are assigned. 

Test administration should be planned in advance. The logistics plan should 
specify: 

« Selection and training of test administrators 

• A test administration schedule 

• Physical arrangements for testing 

If test administrators will need special training, it is well not to wait until the last 
minute to find and train them. A schedule for administering tests should be estab- 
lished, and building principals, classroom teachers, and others affected should be 
consulted beforehand in setting up schedules. The evaluator should check out physi- 
cal arrangements for test administration in advance so that good testing practice 
can be assured. At the least, sufficient space so that students are not tempted to 
cheat will be needed and the testing facility should be comfortable and free from 
noise and other distractions. 

If the performance contract has an auditor he can forestall possible questions 
about testing conditions by observing their administration. He can determine, for 
example, if time limits are properly adhered to, if directions are being given uni- 
formly by all administrators, and if the physical conditions surrounding the test are 
the best possible under the particular constraints of any district. Furthermore, he 
can observe whether students seem to be following directions or answering at ran- 
dom. 

An evaluator can perform the same functions if the project does not have an 
auditor. This kind of observation can be useful in making a judgment about the 
appropriateness of the test for the population to which it was administered. Desira- 
ble testing standards are well known by psychometricians and teachers but the press 



of daily tasks and the limitations of school facilities often make it difficult tc meet 
them. 

The stress on achievement measurement in performance contracting, however, 
makes it essential that LEAs do their utmost to meet the conditions and procedures 
assumed by the test designers. 

During the planning phase the LEA should take steps to insure that the test 
scores wil. represent achievement and not cest-teaching. The LEA and LSC should 
agree on a definition of teaching-to-the test. Wardrop provides a helpful start: 

The items which dually appear in the standardized test . . . are but a 
sample of the items which <x student might reasonably be expected to know' 
and on w'hic'n he might reasonably be tested. It is on the basis of a student's 
performance on this sample of appropriate behaviors that we make infer- 
ences about his level of achievement in the domain of interest . . . Insofar as 
performance on the test may be considered as representative of what the 
student might be expected to do when exposed to that larger collection of 
behavior samples from which the test items were selected, that test perform- 
ance is a valid indicator of his achievement level. 

When the instructional process is such that the particular knowledges or 
skills required for successful performance on the particular test form(s) to 
be utilized are in fact specifically taught, the behaviors sampled in the test 
are no longer representative of the domain to which we wish to generalize. 
Thus, the most crucial consideration in whether '"teaching the test" has 
occurred is whether the instructional content is of such a form as to render 
the test — and consequently normative inferences based on the test perform- 
ance — invalid as an indicator of the general body of knowledge to which 
inferences are to be made . 20 

In a footnote to the preceding quotation Wardrop makes some useful distinc- 



"Preparing" students for a test can take several forms: providing them with 
practice in the test-taking situation by giving them experience with the item 
forms (but not the content) they will encounter on the test, providing them 
exposure to the specific content which they will encounter on the test, giving 
them experience with both the content and form of the test, and coaching 
them on the specific items from the test in the form in w r hich they actually 
appear. The first of these is a legitimate form of preparation in that it tends 
to reduce the contribution of Extraneous, situationally linked factors which 
are irrelevant to achievement in the domain cf interest but which might 
affect performance on the test. The remaining three procedures are illegiti- 
mate (in that they invalidate the test as a representative sample of the 
behavior domain to which inferences are to be made), wdth the last being the 
most blatant and dishonest attempt to invalidate the test and inappropri- 
ately enhance student performance . 21 

Several procedures to deter test-teaching have been used: 

20 J. L. Wardrop. "Was New Century Teaching the Gates-MacGinitie Reading Test in Connection with 
its Providence <R. I.) Contract?,” Opinion prepared at the request of the American Federation of Teachers. 
AFL-CIO. Washington, D.C.. 1971 (mimeograph), pp. 2-3. 



ticns: 



21 Ibid., p. 3. 




* Contractual penalties 

* Secret testing 

* Redundant testing 

Some contracts, such as that in Providence, R- I-, have contained the Educa- 
tional Testing Service definition of test-teaching shown in Fig. 3, and provided for 
substantial financial penalties if the LSC instructs the items included on a standai d- 
ized test. However, the charge by the American Federation of Teachers, supported 
by Wardrop, indicates that this is not sufficient to stop at least suspicion of test- 
teaching, 22 A definition of test-teaching should be in any contract. Penalties ate 
probably also desirable, but probably not sufficient. 

Another approach has been used in the OEO, Virginia, and other programs. 
This is to have an independent organization administer the tests, have it keep the 
tests secret from the contractors, and administer multiple tests in each classroom. 
Three sets of examinations in each classroom plus deletion of the test title and 
publisher's name and prohibition of teachers and other LSC personnel being pre- 
sent, as in the OEO programs, should greatly increase the difficulty of test -teach- 
ing. 22 Unfortunately, it also makes testing difficult; and worse, it discourages feed- 
back during the program that might permit adjustment of the program in light of 
the test results. 24 

For a large program, the OEO approach to preserving test-score integrity may 
be feasible. For a small program it seems too complex. If it is adopted, the i_EA 
should make alternative arrangements for interim checks on achievement and 
feedback to program personnel. 

Another approach is redundant testing — the administration of two different 
standardized norm-referenced tests to the same students on a pre- and post-test 
basis, as was done in the Gilroy program. 25 For contract settlement, only the results 
of the Stanford Achievement Test applied, but as part of their regular Title I proced- 
ure Gilroy administered the Metropolitan Achievement Test. These later data would 
have provided an easy check had there been a suspicion of test-teaching, which there 
was not. Of course, presumably the contractor could have taught to both tests but 
that would have been much harder to do and more likely to be detected. 

-- Ibid. 

2:i For a description of the OEO procedure see Office of Economic Opportunity, op. cit., pp. 14-15. 

24 See P. Carpenter, Case Studies in Educational Performance Contracting: 2. Norfolk. Virginia. The 
Rand Corporation, R-900/2-HEW, December 1971. pp. 61-62. 

25 M. L. Rapp, Case Studies in Educational Performance Contracting: 5. Gilroy. California . The Rand 
Co rporation, R-900/5-HEW. December 1971, pp. 22*32. 
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EDUCATION AL TESTING SERVICE GUIDELINES 



Two items are to be considered the same if 



1. Their wording is identical in all respects. 


Example: A. Which of these is a way to find the circumference 

in inches of a circle with a 6-inch diameter? 

(1) 3x3.14 (3) 3x3x3.14 

(2) 6x3.14 (4) 2x6x3.i4 

B. Which of these is a way to find the circumference 

in inches of a circle with a 6-inch diameter? 

(1) 3x3.14 (2) 6x3.14 (3) 3x3x2.14 

(4) 2x6x3.14 

(Note change in arrangement of options.) 


2. The wording of the stem and the wording of 
the correct response are identical; the 
other responses have been changed. 


Example: A. Same as above. 

B. Which of these is a way to find the circumference 
of a circle with a 6-inch diameter? 

(1) 3.14x3 (3) 3x2.17 

(2) 6x3.14 (4) 2x6x2.1416 


3. The correct response is identical and the 
rr&’iK sense of the stem has been retained 
despite a rrinOT change in wording. 


Example: A. Same as above. 

B. The number of inches in the circumference of a 
circle with a diameter of 6 inches is: 

(1) 6x3.14 (3) 3x3x3.14 

(2) 3x~ ’4 (4) 6x6x3.14 


4. The main sense of the whole item has been 
retained despite the fact that it has been 
restated in the negative. 


Example: A. Same as above. 

B. The number of inches in the circumference of a 
circle with a diameter of 6 inches is ncti 

(1) 6x3.1416 (3) 3x3x3.14 

22 22 

(2) 6x y* (4} 2x3x y* 


5. The main sense of the stem has been retained 

despite a minor change in wording; the correct 
response is identical, but any "incorrect option 
has been changed or omitted. 


Example: A. Same as above. 

B. The number of inches in the circumference of a 
circle having a 6— inch diameter can be found by 
which one of these? 

'!) 3x (2) 6x3.14 (3) 3x3x3.14 


6- The item has been changed from a multiple- 
choice to a true-false format by retaining 
the stem of the multiple-choice item and 
incorporating in the stem one cf the options 
(correct or incorrect). 


Example: A. Same as above. 

B. The number of inches in the circumference of a 
circle with a 6-inch diameter is 3x3x3.14: 

1 | TRUE □ FALSE 



Fig. 3 — Rules for judging whether two items are to be considered the same 



VI. CONTRACTOR SELECTION 



METHODS OF CONTRACTOR SELECTION 

Two basic procedures for selecting contractors have been used in performance 
contracting programs: competitive or formal, and sole-source or informal. In formal 
competitions the LEA circulates a Request for Proposal (RFP) to potential contrac- 
tors, evaluates the responses, and awards a contract if it finds an acceptable 
proposal. In sole-source procurements the procedure is more flexible. Some state- 
ment of LEA objectives is furnished the potential contractor, who prepares a 
proposal. There are iterations until a satisfactory proposal is accepted or the project 
is dropped. 

In some cases informal procedures are used because the contractor has assisted 
the LEA previously. In other cases the LEA has dealt with only one LSC because 
it believed that only that particular contractor had the experience or other qualifica- 
tions it desired. In still other cases, sole-source procurement was chosen because it 
seemed easier or faster than a formal competition . 26 

Formal source selection procedures require that the LEA assess its educational 
needs and define the program objectives, write and advertise a formal RFP, conduct 
a bidders’ conference, and select the winning contractor. Formal competition has 
several advantages: 

• It increases the choice among alternative programs. 

• It increases the probability that the contract will contain desirable protec- 
tive clauses for all parties. 

• It encourages more thorough definition of the program before program 
operation, leading to better planning for program management and evalua- 
tion. 

• It encourages the LEA to support legal research to assure that the contrac- 
tual terms are within the laws affecting the LEA. 

Some hold that a formal competition makes a substantial contribution to pro- 
gram success. Formal competition supposedly forces contractors to seek out the best 

As discussed in Sec. III. in some jurisdictions sole-source contracts may be illegal. 



personnel and the most innovative approaches. Also, it is argued, a firm is more 
likely to give its best guarantees if it is in competition with other firms; in a 
sole-source situation it is more likely to build in ‘'insurance'' factors. The availability 
of a number of proposals facilitates cost comparisons by the LEA. Finally, competi- 
tive procedures minimize suspicions of favoritism that may arise with sole-source 
contracts. 

Others favor sole-source procurements. First, there is some concern that under 
competition a contractor may not be able to propose his "best** system. The contrac- 
tor has to be responsive to the RFP, and therefore may not be able to propose an 
optimum program because it may not respond to the specifications. In an informal 
procedure there can be more interaction and more freedom to make suggestions. 
Many contractors have complained about having to live with costly, unnecessary, 
or counterproductive program features resulting from faulty RFPs. 

A second argument is that formal competitions are expensive. Development of 
the RFP and the evaluation of proposals have required as much as 5 percent of the 
project funds in some programs. 

Proponents of sole-source negotiations also argue that it is difficult if not impos- 
sible to compute cost-effectiveness ratios at the time of source selection. There is 
extensive uncertainty about expected costs and results because the programs must 
initially be developmental. In a competitive situation, it is arguec, prospective con- 
tractors feel they will jeopardize their chances if they are candid about these uncer- 
tainties. The result, it is claimed, is that competitions turn into contests in r broc- 
huremanship.” 



THE REQUEST FOR PROPOSAL 

The purpose of an RFP is to elicit responses from prospective bidders that 
provide, first, a sound basis for selecting a contractor, and second, a bash, for writing 
a contract. The RFP should make it easy for firms to propose innovative systems and 
to take advantage of their peculiar strengths. The more open-ended the RFP, the 
more diverse will be the responses. This virtue can be overdone, of course. An 
inspiring but unspecific RFP may generate responses too disparate to be compared, 
or a competent contractor may be eliminated because he did not understand w*hat 
tne LEA desired. The crux is for the LEA to be specific about what it believes is 
important, but escape entanglement in issues of secondary importance. 

One section of the RFP should provide factual information on the target popula- 
tion, past achievement-test scores, how students in the programs will be designatad, 
their socioeconomic status, the schools involved (organization, size of rooms, electri- 
cal connections, etc.). The more detailed and accurate the information, the fewer 
changes will be required between the program proposed by the contractor and the 
program actually implemented. 

The most helpful RFPs state the criteria that will be used to rank or grade 



contractors' proposals. In some RFPs this has merely been a set of general state- 
ments such as "originality of approach is very important." Other RFPs have pro- 
vided a list of major and minor criteria, with associated numerical weights or points. 
Some RFPs have listed the criteria in some detail but not the weights, and have 
stated that the criteria might be modified. It is important, of course, that ary stated 
criteria and weights accurately reflect the LEA's concerns. Some LEAs have bor- 
rowed, unchanged, other districts' criteria and bid-evaluation weights. This practice 
seems unwise, since no two LEAs are likely to have identical preferences. It is hard 
to see the usefulness of' proposal-evaluation instruments that an LEA does not 
develop for itself and that do not reflect its specific needs. The set reproduced in 
Table 2 is therefore intended to be illustrative only. 

It is unfair to the respondent and counterproductive to the LEA if there is some 
consideration that the school district thinks important but is unstated in the RFP. 
For example, if a school district strongly disapproves of teaching machines, it should 
sav so. If it is important to the LEA to choose its own teach*, rs, it should say :?o; if 
it is not important, the LEA should permit the firm to propose a method for teacher 
selection. 

Administrative arrangements should be spelled out to the extent that they have 
been determined. If management support, evaluation, or audit contractors will be 
used, the RFP should s*y sc and explain their roles. The RFP should explain how 
technical proposals, price bids, or expected costs should be submitted. Past RFPs 
have oflen had the respondent submit his technical proposal in one envelope and 
his pricing information in another, enabling the LEA to appraise them separately - 

Most of the RFPs issued during 1970-71 emphasized two requirements: that the 
bidder accurately describe his approach fur meeting program objectives, and that he 
fully describe his past performance and relevant accomplishments. 

Various lists of prospective LSCs have been prepared by students of perform- 
ance contracting, but they’ change so rapidly they are hard to keep up to date. If the 
LSC is using a management support contractor, he will probably be able to furnish 
a list. The LEA can also work up a list by publicizing the project in trade publications 
or by contacting other LEAs with projects. 

The RFP can be sent out broadcast, or the LEA may decide to limit distribution 
to a selected list. True, elimination of firms from the competition may T lead to charges 
of favoritism, but it may be easier to explain why a firm was not sent an RFP than 
to explain why its proposal w^as rejected. Limitation also reduces the number of 
proposals that have to be evaluated, at the cost of reduced breadth and variety of 
potential programs. 

Frequently, an open conference is held wfith prospective bidders so that ques- 
tions about the RFP can be answered, if a bidders’ conference is held, it should be 
scheduled well before the proposal deadline so that information provided can be 
reflected in the proposal. It is probably wise to have legal counsel or an experienced 
contracting official for the LEA at the meeting so that no legal conflicts arise be- 
tween statements in the RFP and in the bidders’ conference. 



Table 2 



a 

EXAMPLE Or PROPOSAL EVALUATION CRITERIA 

I. Soundness of Approach (2511-3511) 

A. Technical 

1. Theoret i cal /conceptual basis 

2. Pertinent eRjpiric.nl data 

3. Field tested material and techniques 
A. Behavioral psychology basis 

3. Soc io-poli t icai/technical 

1. Will the community accept? 

2. Will the schools accept? 

C. General factors 

1. Degree of nonlabor intensity, i.e., low operating cost 

2. Extent to which instruction is individual! ted 

3. Testing instruments proposed and accompanying rationale 
A. Plan for training local personnel (both consultants and 

para pro fess io-xals) 

5. Motivational techniques proposed 

6. Management and logistic plan 

7. Provisions for quality control and on-going internal evaluation 

5. Range and flexibility of instructional tine per day 

9. Difficulty of transition of aid— year student transfer trom 
Rapid Learning Center to school system 

II. Most Favorable Pricing Arrangement (352-252) 

A. Acceptable methods of co*=t reimbursenjent 

3. Account costs broken into following categories: 

1. Start-up 

2. Capital outlay 

3. Operating, actual, and "opportuni Cy" 

C. Cost per unit achievement for students with different earning profiles 

III. Past Performance and Technical Ability (152) 

A. Relevance of past performance 

3. Verification by check with previous consumers, clients, users* 
associates, etc. 

C. Personnel 

1. Managerial expertise 

2. Background in behavioral science and instruction 

IV. Organizational Commitment (152) 

A. Level of corporate support 

B. Investment of time and other resources in planning proposal 

C. Corporate attitude toward project 

D. If consortium, clarity of lines of responsibility drawn 

E. Extent of other operations and overcommitment 

F. Ability to perform on ’’extras’’ 

1. Social services 

2. Other instructional services 

3. Counseling and guidance services 
A. GED — basic education 

V. Other Factors (102) 

A. Hardware technology 

1. Cost effectiveness of technical, operations 

2. Availability through mass procurement sources 

3. Delivery time and guarantees 

A. Maintenance, re-installation, parts, and repairs 

Flexibility to use various kinds and forms of software and 
conceptual material 

6. Adaptability to modified classroom environments 

Cc y zt7 % octz-Ko z.K Educe t~icK ' The S^-'.'c'iteed StuczerzZ P c 3 2 vnr icz k as 
Ac crccch to Public School Sue is™ Fe ~0 , Educa* 'or. Turnkey Systems, March 1970, 

PP- 19-22 . 
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PROPOSAL EVALUATION PROCEDURES 

In sole-source procurement, proposal evaluation is essentially a decision on 
whether to move to a formal contract, present the LSC with a counterproposal, or 
drop the program. Such a decision can be made in so many ways that no generaliza- 
tion about procedures is possible. In formal competition, it is a common practice to 
designate a committee to evaluate the proposals. Typically, some instrument or list 
of questions about each proposal is prepared. Instruments used in past programs 
have varied. Some have merely been checklists to see whether the LSC has re- 
sponded to all the requirements in the RFP. At the other extreme, some instruments 
have gone deep into the substance of the proposal and required judgments about the 
validity of the motivational techniques used, expertise of the LSC s staff, and other 
complex, qualitative considerations. The list shown previously in Table 2 is a good 
example. 

Source selection instruments have always used many criteria. Clearly, some 
criteria are more important than others and weights or points have to be assigned. 
The definition of the evaluation criteria and their weights is a crucial issue, because 
they will influence the ranking of the prospective contractors- For example, if corpo- 
rate experience is a heavily weighted criterion, experienced firms will enjoy a 
marked advantage over newcomers in the market. Selection of weights can lead to 
questions about fairness and also about the consistencies between selection criteria 
and program objectives. 

Competitive selections are partly intended to prevent favoritism. It is very 
important to designate criteria and weights with scrupulous impartiality and objec- 
tivity*. After criteria and weights have been established, the proposals are scored. 
Sometimes the firm with the top score wins the contract. Sometimes a small sub- 
group. perhaps three firms, are selected and a choice is made from among this group 
bv some higher authority. Sometimes a small group of technically superior contrac- 
tors are selected and a choice among them is made on the basis of the price bids. 

The numerical precision that results from the typical proposal evaluation pro- 
cedure can lead one to forget the underlying uncertainties and qualitative judg- 
ments. The use of formal instruments, committees, and similar procedures increases 
the probability of an objective choice. A quantitative score is obtained, and numbers 
are always impressive. When all is said and done, however, the LEA is dealing with 
promises from prospective contractors, and the criteria and the weighting system 
are themselves qualitative. Ultimately, the choice of an LSC is qualitative despite 
formal procedures that yield quantitative scores. 



DEVELOPMENT OF MATERIALS AND CURRICULA 

Observation of 1970-71 programs indicates that an important and often slighted 
element in contractor selection is the degree of development work to be done. Con- 
tractors’ proposals frequently do not distinguish between materials and procedures 



that they have "in hand” and materials and procedures that they believe can be 
developed . 27 An LEA may reasonably prefer to contract with a firm that has an 
attractive program in a development stage rather than with a firm that has a less 
interesting program that requires little development work. 'Off-the-shelf* materials 
may not be impressive enough to cause an LEA to prefer them to a good development 
effort. Nonetheless, the amount of development work involved would seem an impor- 
tant issue for the source-selection committee to ponder. Important questions are: 

* Do the proposed materials exist? Have they been used in similar pro- 
grams? 

* Is the proposed training program in existence? 

* Do evaluation instruments exist? 

* What is experience and training of on-site supervisors and other per- 
sonnel? 



CONSIDERATIONS IN PROGRAM SELECTION 

It is generally inappropriate to compare several LSCs’ proposals on the basis of 
the potential payment under the payment schedules they present. Each LSC is likely 
to propose to supply a different mix of resources — personnel, materials, equipment, 
and facilities — to implement the program fcr the same number of students. In one 
proposal, for example, the LEA may be expected to furnish all instructional person- 
nel: in another; the LSC may wish to supply and pay for all teachers and paraprofes- 
sionals. In the first case, the LSC may therefore expect a lower rate of return per 
student under his guarantee than in the second case. Moreover, under the perform- 
ance contract, the LSC’s payment depends on student achievement. Since gains are 
uncertain, the final cost is also uncertain. 

Nevertheless, the LEA is naturally concerned about the cost of the program, 
particularly if it extends into a turnkey phase. If the program is to be turnkeyed, 
the LEA will implement the program itself and will incur all of the ‘emaining 
program costs. The cost of the program as part of the regular school curriculum may 
or may not exceed the payments to the LSC under the guarantee. (This point is 
discussed in the Technical Appendix.) Thus, in choosing among various proposals it 
is helpful for LEA planning purposes to estimate what different program configura- 
tions would cost. 

As part of the proposal evaluation process, therefore, a cost-analysis of the 
various proposals is desirable. For the reasons discussed above, this analysis should 
not be limited to the costs of the contract but should cover the total cost implications 
for future years. Some models and techniques for such analyses are in the Technical 
Appendix. 

^ ;:T The Gary experience is a case in point. See G. R. Hall and M. L. Rapp, Case Studies in Educational 
Performance Contracting: 4. Gary, Indiana . The Rand Corporation, R-900/4-HEW, December 19/1. 
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FOLLOW-ON CONTRACTS 

Follow-on contracts — contracts subsequent to the first contract in a program- 
may be let on occasion. Most contracts to date have covered only a single year. 
Most LEAs apparently have assumed that the program would be operated in-house 
after the first year without any direct classroom involvement of the LSC. But what 
is to be done if the LEA decides after the first year that the program merits continua- 
tion but is not ready to be turnkeyed? 

In that event, no matter what its normal source-selection policy, the LEA is 
unlikely to open up the program for competition. Switching contractors in the 
middle of a program is disruptive and creates problems that most LEAs will want 
to avoid. But this fact gives the LSC a bargaining advantage in dealing with the 
LEA . 29 

Several practices can alleviate follow-on contract difficulties. First, options for 
future contracts can be written into the contract. Second, provisions for auditing can 
be required, to give the LEA a basis for negotiation. Third, the parties respective 
rights and ownership of data, materials, equipment, and the like should be defined. 
Then if the LEA does decide to switch contractors it can do so with a minimum of 
disturbance. 

2H The Gary-BRL contract is a notable exception. 

29 Switching contractors is possible (for example, Texarkana switched from Dorseu to EDL for the 
second year of its program) but it has obvious disadvantages. 
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VII. THE CONTRACT 



STANDARD TOPICS IN CONTRACTS 

The ultimate goal of the planning process is a contract that formalizes the 
agreements between the LEA and the LSC. 30 Frequently, the process of going from 
accepted proposal to signed contract has been long and frustrating to those who want 
to get started on the operating program. Often the attorneys involved have advised 
delays to work out ambiguities and contingencies. The difficulties and delays in 
settling many of the 1970-71 contracts indicate that such efforts to clarify and make 
the contract terms precise are worthwhile. 

Past performance contracts have ranged from simple to intricate (see the Tech- 
nical Appendix). Many topics discussed in this Guide might appropriately be in- 
cluded; however, the standard topics are: 31 

* Objectives of program and scope of work 

* Responsibilities, duties, and performance required of LSC 
Responsibilities of LEA 

* Method of measuring performance, and basis of payment and formula for 
payment 

* Procedures for changing the program 

* Procedures for testing 

• Teacher training 

* Teacher administration policies 

• Dissemination of data and information 

• Procedures for visits to the program 

• Successors and assignees 

» Covenant against contingent fees 

• Equal employment opportunity 

• Certification of nonsegregated facilities 

30 Although the discussion relates to the LEA-LSO contract, many of the same considerations apply 
to support contracts. 

31 Adapted from the Texarkana-EDL contract <see R-90C/r HEW). 



• Notice to prospective subcontractors of requirement for certification of 
nonsegregated facilities 

• Rules for deciding whether test items are to be considered identical with 
instructional items or exercises (fie., the definition of test-teaching) 

Most of these subjects are discussed elsewhere in the Guide or are well known 
from standard contracting practice. Two topics, however, require discussion here, 
payment arrangements and provisions for change. 



PAYMENT 

The amount of the payment depends not only on achievement gains but on: 

• Definition of the target population 

« Assignment criteria 

« Attendance requirements 

• Pre- and post-test arrangements 

« Payment formula for pupils not covered by the guarantee 

« Payment formula for pupils covered by the guarantee 

Seldom, if ever, have all students in a program qualified for the performance- 
payment arrangement. In some programs only one-third of the students in the 
program have been tied into the incentive fee system because of the way target 
populations were defined and students assigned, absences, and failures to take tests. 

The target population may comprise all students in one or more schools or some 
subset defined by criteria such as Title I eligibility, IQ above 75, more than two 
grade-level deficiencies on standardized tests, or some other characteristic. 

The criteria used to define the population will presumably affect the contrac- 
tor's materials and techniques. They may also affect the number of students in a 
school eligible for the program or the number of students in a program that come 
under the "guarantee.” 

Assignment procedures have often funneled some students into the program 
who were not part of the target population. For example, a contract may provide for 
instruction of students who have IQs above 75 and are more than two grade levels 
behind in achievement. Students are assigned on the basis of past records, but when 
the pretests are graded and returned, say a month or six weeks after school starts, 
it turns out that some students were less than two grade levels retarded, while 
others fall below the specified minimum IQ level. In these cases: Can or must the 
LEA withdraw the students, even if there is no convenient class for them to go to? 
If the students remain, how is the contractor’s payment for them to be computed? 

Where student turnover rates are high, the LEA may be hard pressed to meet 
its guarantee to provide the LSC with some minimum number of students for a 
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minimum number of days/' 2 Nor is it easy to handle the return of students from the 
program to regular classrooms, or to transfer and adjust the LSC's compensation for 
scudents the LSC wishes to reject for being disruptive or not benefiting from the 
course. 

Some LEAs have essentially argued that the LSC should take "potluck" along 
with the school district, while others have maintained that the program should be 
limited to the type of students for which the LSC's system was developed. Student 
selection procedures, transfer arrangements, and payment provisions for students 
assigned to the program but who do not meet the various criteria for incentive 
payment, should be clearly understood by both parties at the start of the program. 
During the planning process, the LEA is well advised to estimate the number of 
students it believes will meet the criteria selected for achievement payments in 
order to minimize later surprises. 

Despite the drawbacks to a "potluck" approach, it nevertheless seems prefera- 
ble to more selective methods for two reasons. First, in past programs the attempt 
to insure that the students in the program came from the target population created 
substantial managerial difficulties and misassignments occurred despite all precau- 
tions. Second, there is a danger that the program will be regarded as being aimed 
only at "failures," an obvious handicap. 

The performance-payment arrangement usually is limited to students with a 
minimum number of hours of attendance, a number that should be specified in the 
contract. The minimum figure should be substantially less than the district's sche- 
duled number of schooldays per year. Otherwise, strikes, abnormal weather, ab- 
sences, parents moving out of the attendance area, assemblies, etc., may prevent the 
LEA from meeting its minimum guarantees. For example, a minimum of 150 attend- 
ance days might be set if there are 180 days in a school year. This would allow for 
"expected" absenteeism as well as "Acts of God" and school disruptions. 

How is the contractor to be reimbursed for students who do not qualify under 
the performance-payment arrangement 0 The usual way is to treat the students who 
qualify for the guarantee as a sample and use a specified procedure to apply some 
fraction of the payments for t..e sample to all the students in the program. This 
approach, although sound, contains a possible bias. Students who complete the 
course may achieve higher gains than those with excessive absences. Furthermore, 
the LSC may not be diligent about encouraging attendance or test-taking by stu- 
dents who are doing poorly. 

An alternative might be a flat prespecified per diem rate; but this would exacer- 
bate rather than resolve the basic problem with applying the fractional payment 
procedure. It is probably better to extrapolate the payments to the whole group from 
the scores of students with pre- and post-tests, cm less some students fail to qualify 
for reasons that legitimately invalidate the extrapolation procedure. 

When pupils do qualify for the guarantees, the contractors have been paid 
under a variety of formulas linking achievement gains to payments. In some pro- 

32 This was a problem in the OEO experi ment and one factor that delayed settlement of the contracts. 
Office of Economic Opportunity, op. cit.. p. 27. 



grams the pay has simply been a linear function of achievement years of gain as 
measured by norm-referenced tests. In Gary, payment is a function of the number 
of students at or above the national norm on the Metropolitan Achievement Test 
for their respective grades. In a contract in Flint, Michigan, payment was based on 
the rate of ,ain; the LSC sought to double the historical rate of growth of each 
student. A system that was proposed for San Diego, California, wc - d have related 
the contractor's payment to the overall distribution of test scores, comparing the 
LSC students’ achievement score distribution to that of the district as a whole. Th^ 
LSC payment would have depended on how closely its students’ achievement distri- 
bution approached the district-wide distribution- 

Proponents of complex arrangements are worried about the theoretical possib:. 
ity that contractors might concentrate on only a few of their pupils. We question 
whether this concern is of much practical importance. In order for an LSC to 
maximize his profit by concentrating on some students rather than on all, he would 
need a set of highly reliable diagnostic te*ts. He would also have to know just how 
his materials affected achievement gain in light of the initial test score. Given the 
number of students in the typical program and the state of the art of testing tech- 
nology and learning theory, it is doubtful that any contractor can rationally decide 
to favor one student over another. This situation may change as testing and instruc- 
tional systems improve, but at present there seems to be little danger that the use 
of simple payment arrangements may lead to some students being ignored. 

The results in the 1970-71 school year, for the most part, argue against expect- 
ing very large increases in achievement test results. Goals on the order of 1-7 or 2.0 
achievement years per year represent a very substantial increase in the rate of 
growth of the students’ achievement- Less ambitious objectives seem more appropri- 
ate, given the history of performance contracting thus far. The Flint, Michigan, 
approach of defining the objective in terms of doubling each student’s historical ra^e 
of growth has the attractive quality of emphasizing the implied change in each 
student’s achievement. 

The OEO, Texarkana, Virginia and some other programs have included pay- 
ments based on criterion-referenced or learning-mastery tests. This is an attractive 
approach since it defines payments in terms of the objectives of the program rather 
than the wider objectives reflected in standardized tests. As discussed elsewhere, 
however, the state of the art of criterion-referenced testing leaves much to be 
deGred. Payment arrangements for criterion-referenced test results also pose diffi- 
culties because one must be assured that the students actually mastered the objec- 
tives in the program, not prior to it, and there are no real norms against which to 
compare results- 

Other payment arrangements can be conceived. For example, at one point in 
the Texarkana project consideration was given to making the payment to the con- 
tractor a partial function of the dropout rate of the students :n the program- Again, 
the problem is how to relate the contractor’s achievement to some norm. 

The basic theory of performance contracting implies using the payment ar- 
. ar.gements to motivate the contractor. Therefore- the payment arrangements 
should reflect the LEA’s objectives. On this criterion the San Diego approach is 
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appealing; the district wanted to lessen the differences among its schools and it 
proposed to reflect this goal in its payments to contractors. There are three other 
considerations, however: 

• Complex formulas are often hard to monitor 

• Complex goals are hard for the public and sometimes the project personnel 
to understand 

• Incentive clauses provide motivation only if the contractor has the means 
to achieve the bonus or avoid the penalty 

If, for example, a special bonus were offered for all students with IQs below 75 
who achieved 1.5 grade levels of gain, it would be possible to identify such students 
but the offer would be empty unless the contractor had special techniques for train* 
ing such students. 



CHANGE AND SETTLEMENT ARRANGEMENTS 

Because each LEA has unique charac .eristics, a program developed for one LEA 
will have to be adapted to fit another. Performance contracting technology is not yet 
standardized. As discussed earlier, nearly all programs are developmental; they 
change during the school year to cope with unforeseen problems and take advantage 
of opportunities for improvement. The result is that the LEA and LSC wall often 
have to settle on the basis of a program quite different from that initially envisioned 
in the contract. 

Change itself may be a favorable aspect of performance contracting. Perform- 
ance contracting encourages teachers to find out how they are doing and revise 
unsatisfactory materials and procedures. This is a healthy motivation, not always 
found in conventional school programs. If change is to be encouraged, however, 
appropriate contract provisions are required. 

Multiyear projects have a great advantage over single-year projects because it 
always take time to implement a new program. The true significance of the first 
years evaluation may lie almost entirely in its effectiveness in contributing to 
program implementation as contrasted to its assessment of program success. In the 
Gilroy program, for example, the program staff soon decided that a change in the 
schedule was necessary but the other teachers resisted disruption of their classes, 
resulting in a six-month delay in implementing the changes. Appropriate contract 
provisions might have avoided the impasse, to the benefit of both parties. 

Perhaps the best example of the problems that can arise because of unforeseen 
circumstances is the 1969-70 Texarkana program, which was involved in the teach- 
ing-to-th e-test dispute. This contract has yet to be settled. 

It is not easy to write contracts that appropriately provide for possible changes. 
If contracts are couched in general language., relying on good faith to work out 
matters as they arise, obvious advantages or disadvantages may ensue. On the other 
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hand, if contracts try to cover all contingencies, complexities result that have 
delayed the start of some programs. 

It is probably impossible to anticipate all possibilities anyway. The best ap- 
proach seems to be a mixed strategy of writing clauses for those contingencies that 
can be easily envisioned and building in procedures for amending the contract to 
cope with unforeseen events. It is particularly important to specify what parties may 
initiate these procedures and who must accede to changes. The OEO contracts are 
good examples of this mixed strategy. 





Part 3 



Program Operations 



VIII. organizing and monitoring the program 



This section discusses the major activities required to get a program under wa> 
and maintain it in operation: 

• Selection of schools 

• Personnel selection 

• Teacher training 

• Student selection and assignment 

• Monitoring the program 

• Extending awareness of the program 



SELECTION OF SCHOOLS 

Preferably, the school or schools that will house the program will be ^elected 
during the planning phase™ but selection has often been postponed until the begin- 
ning of the operational phase of the program. In deciding on a school, five considera- 
tions are particularly important: 

• Population characteristics 

« Turnover rates and absents dsm 

* Size of school 

• Support of building personnel 

* Accessibility 

If the program is funded by categorical aid (such as Title I). the school selected 
will have to meet the applicable criteria. If the program ;s financed b> general fun s 
or if several possible sites meet the funding requirements, the range of criteria can 
be widened. The other considerations— turnover rates, size, etc.— may. then deter- 
mine the choice. If not, the choice should be made with validation and evaluation 

:,:l The designation of comparison or control schools was discussed in Sec. V. Planning the E\alua- 



in mind. The district will presumably want to generalize the results and the schools 
should he selected to aid in generalization. If the focus is on compensatory education, 
a district might pick a school with a particularly poor educational situation: to test 
the learning system on a turnkey basis, a more typical school might be a better 
selection. In short, the selection of a school ideally will reflect the policy issues to 
be addressed in the evaluation. 

The importance of turnover and absence rates has been previously discussed. 
There are cost and other advantages to having a relatively large group of students 
in a program (200 to 400 cr more*. If the program is sited in a small school, all the 
students in some grade or grades may have to be assigned to the program. Doing so 
simplifies scheduling and avoids distinguishing "fast from slow students. It may 
make it difficult, however, to limit the program to students with severe learning 
difficulties or other particular characteristics. 

The support of the principal, teachers and community may be a factor. Like any 
new program, a performance contract will require changes in school procedures and 
it is an advantage if those affected by the program are interested in it. 

If the school system is within a major metropolitan area served by reliable 
transportation, it may not be necessary to consider accessibility when choosing 
schools for the program. If’ it is difficult for participants to reach the schools, how- 
ever, travel constraints may discourage them from doing their job well. Remoteness 
from the school has contributed to inadequate participation in the past. 



PERSONNEL SELECTION 
Teachers 

Teachers are- the key element in any learning system. One implication from 
past experience is that teachers should be involved in planning the project. Another 
implication is that the program management should insure that teachers are imple- 
menting the learning system according to plans. In several past programs knowl- 
edgeable officials complained that teachers were not implementing the system as 
intended, and the results, therefore, were not appropriate measures of what could 
be done. 

Another aspect of this problem is that if teachers are to implement the system 
they must have the training, materials, and facilities required and receive them on 
time. Ir several ' \si programs, the promised logistics support was late or insuffi- 
cient. Classrooms /ere not remodeled in time for the opening of school or they were 
too small f< ' the armed program, and new materials did not arrive on time. 

"Off-the -shell systems that can be simply plugged into any school's program 
and work successfully are apparently nonexistent. Program managers must there- 
fore be alert to adapt the program on the basis of feedback on results. The best source 



for much of the feedback is the teachers. Teacher-initiated changes should be under- 
stood and approved by all the project management, however. 

If the LSC and LEA take different approaches to rules and procedures, teachers 
mav be caught in the middle with dual responsibilities and loyalties. The LEA will 
be wise to set general guidelines about the LSC s authority, if any, to modify regular 
rules/ 14 

In most programs, teachers and the LSC have established harmonious and 
effective relationships but in a few programs personnel changes have been neces- 
sary. Some LEA executive should be empowered to move quickly and resolve mat- 
ters when they first arise. In at least one past program the changes involved teacher 
transfers: in at least one other program, however, the LSC staff was revamped. 

In some programs the LSC has been assigned teachers by the LEA. Often, 
however, the LEA has drawn up a 1 ist of candidates and the LSC has selected its stafi 
through interviews. This method helps to assure that the contractor will ha\e 
teachers he can work with and who will be willing to take his direction and support 
his effort. It does, however, introduce another element of nunrandomness into the 
“quasi-experiment” because the LEA is likely to designate superior teachers and the 
LSC is able to pick and choose. 

Criteria for teacher selection might include: evidence of interest in and support 
for the program, past experience with similar students and subjects, and evidence 
of’ ability to adjust to new situations, 

Paraprofessionals 

Paraprofessionals are usually hired especially for the program. Both the LEA 
and LSC often interview the people concerned. Usually an effort is made to obtam 
residents of the local community. 

Many past programs have made effective use of paraprofessionals. The stress on 
individualized instruction creates the opportunity for teacher aides to be involved 
in classroom management and free the certificated teacher from many routine 
duties. The certificated teacher can concentrate on diagnoses, individual tutoring, 
affective counseling and other specialized activities. The specialized role that the 
paraprofessional can plav implies that special care be given to their recruitment and 
training. 

Frictions and personality* clashes, of course, sometimes arise. A relatively sim- 
ple tactic to decrease this possibility is encouraging program teachers to help select 
aides. Teachers may already know people they would like to work with and can 
propose candidates. 

Unionization of teaching paraprofessional is unusual, but a large-scale per- 
formance contract may involve so many that unionization may become an issue/ 15 

;t “ To illustrate, jealousy arose in one program because the LSC permitted smoking in the learning 
center v.'hile the LEA did not permit it in regular classrooms. 

:!5 See Hall and Rapp. op. c:t., pp. 86-87. 



Principals 

For the host school, a performance contract inevitably engenders new schedul- 
ing require ‘s and special requests for facilities and services. If the principal is 

"left out” r y understandably view performance contracting as merely a source 

of added work. In programs where the principal is outside of the direct chain of 
command, special attention by both the LEA supervisor and the LSC director is 
required to keep the principal in the decision "loop” and to make sure that his 
interests are understood and resoected. 



LSC Director 

The on-site LSC director occupies an important and difficult position. He may 
be an expert, but he is an outsider. In past programs, if he was professionally 
qualified, flexible, and able to get along with teachers and parents, resentment and 
fear died down. If he was abrasive or maladroit, problems resulted. It may be 
desirable to include a clause in the contract permitting th c j^EA to recommend 
replacement of the LSC director if problems cannot be solved in a reasonable period. 
It will be judicious for the LEA to interview and approve the prospective on-site 
contractor personnel. 



LEA Supervisor 

A good LEA supervisor can be the key to program success. He should be a 
decisive official with sufficient authority to permit him to resolve most questions 
quickly on his own authority. Implementing a new and innovative program requires 
cutting much red tape and overcoming many obstacles to change. This can only be 
done if the program is backed by an LEA official with the ability, professional 
respect, and authority to get things done. 



TEACHER TRAINING 

LSCs typically provide a week or two of training for certificated personnel 
before the program begins and some in-service training while the program is under 
way. Paraprofessionals typically receive somewhat less training. 

In the press of getting a program started it is often difficult for both the LEA 
and the LSC to provide good preprogram training. In-service training may also be 
slighted because of other demands on teacher and contractor time. This neglect is 
understandable, but it is unwise. Training programs are important to both parties. 
The LEA and LSC should review the preservice training program to make sure that: 
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• There is sufficient time before school opens (at least a week) to familiarize 
the trainees with the new approach. 




• The program will instruct trainees in using the materials and methods in 
teaching students in the target population. A good sample of the instruc- 
tional materials, diagnostic test, and other components of the instructional 
system should be available to trainees. 

The LEA should make advance arrangements for the facilities and equipment 
that will be needed. Positive inducements should be offered to trainees for attend- 
ance at trainir sions. In some programs the LSC has paid overtime for attend- 
ance. This pra provides a slightly higher yearly income for program partici- 

pants without deviating from the established school district salary scales. It also 
enhances teacher motivation to attend the training sessions. Considering the impor- 
tance of training, overtime payments would seem to be a useful practice and a 
reasonable expense. 

Because performance contracting programs are developmental, it is particu- 
larly desirable to provide for in-service training throughout the program. Teachers, 
supervisors, and contractor personnel ohould have frequent opportunities to discuss 
and resolve the problems that are bound to arise and to share particularly useful 
and rewarding experiences. In past programs, teachers often complained that they 
had no one to talk to after the program was well under way, and sometimes replace- 
ments for teachers who had to leave the program received no training at all. Regu- 
larly scheduled in-service training sessions, say as often as twice a month, would 
alleviate such problems. Again, some inducement for attendance would be desirable. 



STUDENT SELECTION AND TRANSFERS 

Student selection and transfers have been the major operating problems in 
several past projects. Three types of problems have arisen: 

• An undesirable image for the program created by the selection criteria 

• Students enrolled in the program who are not in the target population 

• Conflict with other school classes or programs 

If student? obs* -i e that only academically inferior students are consistently 
chose - to be m the class or if they hear teachers characterizing the program as 
having been designed only for the "slow learners,” an image of the program as tne 
"dui my class” may result One way to avoid this is to enroll entire grades in the 
program; another is to enroll students with a range of capabilities. 

If the times that the performance contracting classes are offered conflict with 
other course offerings students may not want to participate in the program. This 
is particularly likely to arise at the secondary level, where students have some 
freedom in course selection. This problem can be avoided by paying careful attention 
to scheduling details or by substituting program subjects for subjects in the regular 
curriculum. The latter tactic is sometimes possible for particular subjects but not 
for others. Program math is often substitutable for regular English. 



Students are often selected on the basis of regular district tests administered in 
the previous year. Transfers and late entries naturally complicate preparing the 
roll. What is more serious is that when the initial pretest for payment purposes or 
the original diagnostic tests are administered, it may well turn out that a number 
of students do not meet the selection criteria. Because of this, in some past programs 
there has been a considerable time lag between the opening of school and the final 
assignment of students to the program. This time lag has meant that students are 
reassigned late from the program to the regular school classrooms, to the unhappi- 
ness of the students and teachers. It also means that replacements for the reassigned 
students must be found and these must be taken from regular classrooms and 
assigned to the performance contracting program after the pretest. This may create 
confusion and disenchantment with the program. It also makes it difficult for such 
students to receive both a pre- and post-test for payment purposes and to receive the 
minimum number of days of instruction specified in the contract. 

To minimize these problems, it would appear desirable to schedule students 
during the summer with the greatest care possible. The expediting of the prepara- 
tion of the final roll should have first priority at the start of school. If assignment 
is to be effected by the initial tests, these should be administered promptly and the 
results scored and evaluated at once 

Some of the early programs had provisions for transferring students from the 
program when their achievement met specific criteria such as attainment of some 
target gain or accumulation of excessive absences. If provisions are made for remov- 
ing studei/ s from the program before it has run its course, they usually must be 
replaced so that the contractor will have enough students to work with. Also, if 
student exit is determined on the basis of test scores, then students should not be 
overtesved simply because they repeatedly fail to show the gain the teacher thinks 
they have made. In contrast, inadequate provisions for testing late entrants or early 
leavers may menn that contractor payment is based only on a fraction of the pro- 
gram participants. Exits at irregular times also put students into unfamiliar class- 
rooms part way through the semester, which may be detrimental to their standing 
in the subject. The best solution to all of these problems may be to plan fc'. entrance 
and exit of students at specified times, such as between semesters. 



MONITORING THE PROGRAM 

A system of cognizance and control over the program is essential to: 

• Maintain the LEA’s legal and administrative control 

• Insure that teachers adhere to the program as designed 

• Provide early warning of potential conflicts among program personnel 

• Minimize scheduling and management conflicts between the program and 
the rest of the host school’s program 



56 



Permit resolution of operating questions at the operating level through 
interactions between the LEA and LSC managers 
Identify program improvements 



Selecting Monitors 

Monitors have included: 

• Full-time LEA managers 

• Ad hoc executive reviewers 

• Local building administrators 

• Management support groups 

• Evaluators 

In the OEO programs each LEA appointed a full-time employee to supervise the 
programs. This procedure provides a single focus of responsibility and someone with 
time to become familiar with the program. Problems can be resolved promptly. This 
approach is expensive, however, and the monitor can become too involved in the 
program to be objective about it 

In some programs supervision has been a collateral duty of an assistant superin- 
tendent or other high echelon LEA executive . 36 If action is required, it can be 
promptly initiated without having to move up the chain of command. An executive 
may be too busy, however, to devote much attention to the program. 

In other programs, a principal or assistant principal in the host school has been 
given monitoring responsibilities . 37 Principals are in a position to observe the pro- 
gram continuously but they have many other responsibilties. Conflicts of interest 
may also occur in coordinating the operation of the program and the school. 

Monitoring is a possible role for a management support group. The MSG is in 
a neutral position from which it can act as a go-between. Because MSG services are 
expensive, however, there is a tendency not to use them on a continuous basis; it may 
be some time after a problem has arisen before the MSG is able to deal with it. 

Another approach is to utilize the evaluator as a monitor. As an adjunct of his 
evaluation activities the evaluator can monitor the management and process as- 
pects of the problem and take an active role in suggesting improvements and resolv- 
ing difficulties . 38 There are two drawbacks, however. The evaluator may not be in 
the district very often. Second, some evaluators perceive a conflict between pro- 
viding interim guidance and their need to keep program data confidential so as not 
to contaminate the final validation of achievement gains. 

;J6 The 1970-71 Gilroy program is an example. See Rapp, op. cit. 

U7 The CMES program in Grand Rapids is an example. See G. C. Sumner, Case Studies in Educational 
Performance Contracting: 6. Grand Rapids , Michigan. The Rand Corporation, R-900/6-HEW, December 
1971. 

:,H In the 1970-71 Texarkana program the same organization had both an evaluation contract and a 
management support contract, with excellent results. See Carpenter, Chalfant, and Hall, op. cit. 






Records and Observation Reports 

To maintain cognizance and control, four types of records and data are required: 

• Tests and other measurements 

• School district records of attendance, transfers, etc. 

• Records of LSC, MSG, and evaluator activities 

• Observations and unobtrusive measures 

Basic sources of data are the results of the interim and final test given by the 
evaluator. These data can be useful not only for validation and evaluation of the 
program but for program management. 

School district records such as attendance data, disciplinary referrals, trans- 
fers, costs, and like information provide evidence of how the program is progressing. 
Unfortunately, in some past programs such data have been sporadically collected 
and even lost. It is best to devise a formal system for capturing and preserving 
relevant school district data. Examples of important records of program participants 
are: LSC records of in-service training, visitor logs, and minutes and attendance 
figures for teacher meetings and parents' meetings. In the press of meeting daily 
commitments it is extremely likely, on the basis of past experience, that such records 
will not be kept up to date. The LEA might periodically check or collect the records 
to ensure having the data. Alternatively, an MSG or outside evaluator, if one is used, 
might audit the records. In any event, if history is a guide, without special attention 
LSC data available to the LEA will probably be fragmentary. 

Unless the data contribute directly to the collector's primary task (as data on 
student progress contribute to the teacher's instructional program), special steps 
will be needed to assure that the data collected will be complete and accurate. 
Therefore, in some instances it may be wise for the evaluator, auditor, program 
director, or whoever else is given the responsibility, to make unannounced spot- 
checks of daily records. But the same data should not be entered on different forms 
merely to satisfy the needs of different agencies. The forms should be kept as simple 
as possible. 

If an MSG or outside evaluator is used, it is important that what they learn 
during their visits be captured by the LEA. A useful device here is the "contact 
report” as used in the OEO, Texarkana, and some other programs. This is a printed 
form that the MSG or evaluator fills out whenever he deals with someone in the 
district and learns about some matter that should be brought to the LEA’s attention. 
Trip reports and "debriefing meetings” can also be used. 

Finally, as a source of data there is no substitute for direct observations, conver- 
sation with teachers and other program participants, discussions with parents and 
students, and similar formal contracts. 



EXTENDING AWARENESS OF THE PROGRAM 



Every past program has sought parental and community involvement. It is 
intuitively obvious that parental support for a performance contracting program 
can be extremely helpful. Parents may be naturally supportive . 39 In many communi- 
ties, however, parents show little interest in school programs and may be cynical or 
even hostile. If confronted by apathy or hostility, the LEA will need to take positive 
steps to enlist parental support and involvement . 40 Such efforts may have both 
short-run and long-run benefits to the school in changed community attitudes. They 
will also provide opportunities for program personnel to explain new’ features of the 
program that may puzzle or disturb parents. 

To develop community awareness, the district will probably have to mount a 
public relations program. Interest in the program has two -ides. It creates a require- 
ment for the LEA to handle public and professional relations, and it creates an 
opportunity for the LEA to generate public support. 

A performance contracting program is likely to lead to many requests for visits. 
Program personnel in past programs have found themselves hosting many teachers 
from other districts, reporters, researchers, and other interested people. This re- 
quires much time from the program staff, and in extreme cases has been disrup- 
tive . 41 Special visiting days have been established in some programs. Any program 
should have a set of established procedures governing who can visit the program 
under what conditions and at what times. 

However, teachers not in the program might well be encouraged to observe it 
in action and be given the time and means to do so. If visits must be limited, teacher 
visits should be given top priority. Teachers should also be encouraged to sit in on 
in-service training or consulting sessions to learn about the program's good and bad 
features. 



39 Parental support was extremely important in BRI/s Gary program. See Hall and R * PP ’ ° P * CR * 
« In Greenville, South Carolina, the manager installed a telephone in the center and ^called the 
parents of students when they had a particularly meritorious day. One also make f u ^ e g 0 ™.® 

by teachers or other program participants, although we know of no performance contracting program 

1969-70, program persannel had to devote a great deal of time to showing 

visitors around, speaking at symposia, and so forth In Gary, ^thtA^ffertetT the* program^The 

Banneker School observing and interviewing teachers so frequently that ltattected the p g 
evaluator recommended at the end of the first year that such exposure e lmi e 



Part 4 



Evaluation 



IX. VALIDATING ACHIEVEMENT AND SETTLING THE 

CONTRACT 



Experience has shown that contract settlement may generate a series of legal 
moves and countermoves, claims and counterclaims, with bitterness and disenchant- 
ment on both sides, particularly for programs in which measured student gains are 
appreciably less than those "guaranteed” by the contractor. It is not uncommon for 
contract settlement to take six months or more. Many of the contracts involved in 
the OEO experiment, for example, were still unsettled in January 1972. 

Since data on program results are usually not released until final contract 
settlement, there is often a long period in which the LEA is unable to respond to 
requests for data. This delay can pose public relations difficulties. 



VALIDATION OF STUDENT ACHIEVEMENT 

The purpose of the validation is to assure that contractor payment is deter- 
mined objectively and accurately, and as specified in the contract. Two situations 
may cause disputes about the appropriateness of the tests used as measures of 
student learning. On the one hand, the subject-matter content of the LSC’s program 
as actually implemented may have had a much weaker relationship to the content 
of the measuring instruments than the LSC had originally anticipated. On the 
other hand, the content of the test and the instruction may have matched too well, 
i.e,, the LSC may have taught the test. 

Several methods for guarding against teaching the test were previously dis- 
cussed, but even if they are used the suspicion of test teaching may still arise. To 
refute or support such suspicions, the evaluator or auditor should analyze how the 
test items correspond to the materials used in the instructional program. (This 
further supports the need for continuous monitoring of the program during actual 

42 Office of Economic Opportunity, op. cit., p. 27. 

43 This appears to have happened in the Norfolk program. See Carpenter, op. cit. 



instruction.) Such an analysis also will be useful for specifying areas where the 
program is particularly strong or weak. 

A similar analysis will reveal whether the test adequately sampled the content 
of the instruction. For example, if students spent most of the school year learning 
skills that are not tapped by the test, the test will not measure program effective- 
ness. 

Matching test contents and the instructional program determines whether the 
test is a valid measure of what was to be learned. Whether it is an accurate measure 
may require further analysis. Newly constructed tests often contain items that elicit 
wrong responses because they are ambiguous or misleading. The reliability of in- 
dividual test items and of the test as a whole can and should be determined by 
analysis of student responses to each item and to sets cf items. 

"Murphy's Law" — if anything can go wrong, it will — applies to every stage of 
the validation. The following items should be checked: 

• Information on test booklets or answer sheets that identifies individual 
students’ test scores 

• Characteristics of students jre if they fall within contract specifications 

• Days of attendance 

• IQ, if the contract specifies a minimum level 

• Existence of a pre-test score or a post-test score or both 

• The level of the pre-test score 

• The publisher of the pre-test and post-test (if several tests are used it is 
possible that some students could be given pre-tests and post-tests from 
different publishers) 

• The actual numerical computation of payment 



CONTRACT SETTLEMENT 

After the tests have been administered, scored, and analyzed, the evaluator 
must identify the students who qualify for the performance guarantee under the 
terms of the contract as well as those for whom other special contractual arrange- 
ments may have been made. There may even be a group of students for whom the 
contractual payment schedule is ambiguous. The evaluator must resolve such issues 
through negotiation with the LEA and LSC or by contract interpretation. 

Finally, the evaluator must compute the payment due the LSC by the formulas 
applicable to each group of students falling within specified contractual arrange- 
ments. Such computations are rarely complex, but errors can occur in even the 
simplest of situations. Computation of payment may also require interpretation of 
the contract . 44 Also, in programs where there have been a number of changes, it may 

44 Fur instance, the Gary-BRL contract states that "at least a year’s advancement in reading and 
mathematics will be achieved.” In June 1971, the question arose as to how to define "a year’s advance- 
ment” and relate it to the test score. Price, Waterhouse and Co. was hired to compute the payment, and 



not be obvious how the original intentions should apply to the new situation. 4 " The 
two parties might negotiate an interpretation . 46 The Gary app r oacli of hiring an 
auditor to do the interpretation and calculation is an appealing alternative. A 
management support group or independent evaluator may also provide the interpre- 
tation. 

it interpreted the clause to mean that a student had to achieve a month’s advancement for a month’s 
instruction in both reading and mathematics. 

45 For an example of program changes that affect settlement computations, see the report on Gilroy- 
Rapp, op. cit. This program was converted from one with variable exit times to one in which all students 
remained in the program the full school year. 

Contract settlement details of actual programs are frequently confidential. 




X. ASSESSING THE PROGRAM 



The purpose of assessing the program is to provide a basis for decisions about 
its future in the school system. For ruch decisions, the program’s effects on the 
students’ cognitive growth ire clearly important, but other information is also 
needed. Information gathered during the course of the evaluation should assist in 
settling such issues and should suggest needed changes. Finally, the options for 
program implementation should be compared with one another and with other 
alternatives that may be available to the school system. 



EVALUATION 

Program assessment consists of (1) evaluation and (2) the choice of alternatives. 
The evaluation should provide a description of the development of the program 
within the school system in order to answer such questions as: 

* What was "the program” in fact? 

* How long was "the program” actually in effect hi the classroom? 

* Did start-up problems reduce program effectiveness? 

* What were the major obstacles to implementation? 

* How did attitudes toward the program change as it was implemented? 

* What program features seemed to generate the most enthusiasm or to 
work the most smoothly? 

A major function of the evaluation is to gather quantitative data on program 
effects. (Some suggestions on the kinds of data that should be gathered were set forth 
earlier.) Analysis of these data will reveal whether the program had effects and how 
extensive they were. As noted previously, the use of control groups is very helpful 
at this point. In most situations, the data warrant only relatively simple analysis. 
Elaborate analyses of incomplete or inaccurate data, or sophisticated comparisons 
of data that are not really comparable, are a waste of time and misleading. 

The significance of program effects on student learning to program assessment 
is obvious. (Since numerous passages in this guide have discussed the measurement 

66 68 



of cognitr/e achievement, we need not treat it further here.) The analysis ot test 
items proposed in Sec. IX as part of the validation will also provide useful data for 
program assessment by illuminating the weak and strong parts of the program. This 
wilJ point the way to whatever changes in content or emphasis are desirable. 

The evaluation should also assess significant nonquantifiable effects of the pro- 
gram, such as changes in degree of acceptance or support for the program on the 
part of teachers, administrators, parents, community groups, teacher unions, and, 
of course, the students themselves. Many of these effects can be captured by observa- 
tion and interviews. Changes in the classroom environment are also important. 
Other intended or unintended outcomes may result from the program. The evalua- 
tor should describe any anticipated or unanticipated results because unanticipated 
side effects may be key considerations in decisions about the future of the program. 



CHOICE OF ALTERNATIVES 

The evaluation should provide direction for the design and comparison of alter- 
native program configurations to be considered for the future. One such alternative, 
of course, is for the school system to revert to its prior practices. Since this alterna- 
tive was inadequate in the past, it is merely a baseline forjudging the incremental 
effects of other alternatives. 

Another alternative is to adopt an approach entirely different from perform- 
ance contracting — some other type of remedial program, for example. This alterna- 
tive would be attractive if the program just completed were patently unsuccessful 
or unaccep table to the school system, if the educational need remained as pressing 
as ever, or if some potential alternative had been tried in another school system and 
had had outstanding results. The disadvantage, of course, is that the school system 
would have to go through the implementation process all over again. 

There will also be a set of alternatives based on the program just completed. 
Changes to the program might be made in several areas: 

• Content . Should subject matter be deleted or added? Are changes in em- 
phasis needed? 

• Students. Should the program be extended to more students? Should a 
student population with different characteristics be included? 

• Staffing. Should the mix of students, teachers, and paraprofessionals be 
changed? Should the star: be acquired, trained, and paid in the same ways? 

• Management. Should the LSC continue to have responsibilities for in- 
struction? If so, should he continue to work under a perftm *ance guaran- 
tee? 

• Evaluation. Should the evaluation be conducted as before? 
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Although many alternatives could be generated by all possible answers to the above 
questions, probably only a few will need to be considered. 



One of the options is to turnkey the system, i.e., to operate the program as part 
of the LEA’s regular program without direct involvement of the contractor. The 
term turnkey ’ is widely but loosely used in discussions of performance contracting. 
When the future of the program is being decided, however, the LEA will have to be 
precise about the operational meaning of the term. Turnkey has been used to refer 
to three very different types of adoption: 

• LEA conduct of the program exactly as the contractor had managed it, 
using the same materials and facilities and instructing about the same 
number of students. 

• Retention of the same procedures and materials, but with more students 
involved. 

• General adoption of the basic materials and techniques for use within 
conventional classrooms. 

Each of these "turnkey” options has very different implications for training, cost, 
and evaluation. 

After the educational process and outcomes have been assessed, resource re- 
quirements should be analyzed. The objective is to estimate the impacts of the 
alternative programs on the district’s resource inventories, personnel, and cash flow. 
This will provide input to financial planning and resource management on the one 
hand, and more information for designing the scope of the learning programs on the 
other. Such a planning exercise should have a time horizon that spans the initial 
phase-in period and any anticipated major modifications. It should also include at 
least the first year in which costs and operations are expected to level off. 

The general scheme is to collect certain resource information on the candidate 
programs, incorporate this information into a design for the planned implementa- 
tion of the programs, and then arrive at schedules of resource requirements. Re- 
sources that are to be supplied by the contractor, or are otherwise available without 
additional cost to the district, are deleted to obtain a list of the incremental resources 
that the district must acquire. 

We suggest a seven-step analysis: 

1. Summarize the resource requirements of the candidate programs. 

2. Derive resource factors. 

3. Describe the scope of the planned implementation of the programs. 

4. Project resource needs for the planned programs from steps 1 and 2. 

5. Subtract, from district costs, those resources to be supplied by the contrac- 
tor or to be otherwise available without cost. 

6. Summarize the incremental impacts on resource inventories and staffing 
levels on a year-by-year basis. 

7. Display summary program costs on a year-by-year basis. 

Such an analysis is illustrated in the Technical Appendix. 

Expected costs must be related to educational effectiveness as measured by 
indicators that describe the learning (or other stated objectives) of a program. Some 



programs are directed specifically to goals other than improvement of student 
achievement in such subjects as reading and mathematics. For example, the pro- 
gram may seek to change the students’ self-images. Objectives such as these are 
usually thought of as fostering the attainment of student achievement as a primary 
objective, but the causal relationship may go in both directions. In any event, if the 
prograr., I^votes resources specifically to attain such ends, measures of the extent 
to which these ends have been attained belong in the set of indicators of program 
effectiveness. Finally, the attitudes of parents, teachers, and the community at large 
toward the program may suggest peripheral benefits and can be used to choose 
among programs of apparently equal teaching effectiveness. 

A program’s effectiveness can be summed up only by means of a set of measures 
and indicators. To choose among alternative programs, the planner must judge the 
relative importance of these indicators as they apply to the students and schools in 
question. For example, reading improvement may be a driving goal in inner-city 
schools but only a mild concern in schools in upper-middle-class neighborhoods. 
Similarly, changes in teacher status may cause difficulties in school systems with 
strong teacher unions but may be accepted fairly readily elsewhere. 

Given a set of alternative programs accompanied by descriptions ol their re- 
source requirements and their effectiveness, they can be compared on the oasis of 
their relative returns. Such a comparison is often termed a cost-effectiveness analy- 
sis. There are two basic ways to conduct such an analysis. One is to specify the level 
of effectiveness desired and then compare the costs of alternative means to achieve 
it. The other is to specify an acceptable cost level and compare the relative effective- 
ness of programs whose costs cluster around that level. In the educational context, 
however, the answers can rarely be more than rough approximations. It is seldom 
possible, in education, to compute a truly meaningful cost-effectiveness ratio — that 
is, an accurate one derived by statistical methods from an abundant data sample. 
Strictly speaking, given a budget constraint (the usual situation in education), such 
a ratio can be computed and used to rank alternatives only if numerous levels of 
effectiveness within the same alternative are available for consideration, up to the 
limit of the budget. 

In analyzing instructional programs, the fixed-budget approach (the second 
method described above) is probably the most helpful. That is, the LEA should say, 
’'We have $100,000 to spend on 350 students. Which program will give us the greatest 
effectiveness?” 

In assessing a performance contracting program and deciding about its future, 
these two salient points should be kept in mind: 

1. A broad range of possible program outcomes — favorable and unfavorable 
— should be examined in addition to cognitive test scores; and 

2. The cost and effectiveness of the possible implementations of the perform- 
ance contracting program should be compared relative to each other and 
to the situation without performance contracting; 
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